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Abstract 

The  purpose  of  this  thesis  was  to  assess  user  requirements  for  an  automated 
information  system  to  support  programmed  depot  maintenance  (PDM).  To  accomplish 
this,  the  Integrated  Technical  Information  for  the  Air  Logistics  Centers  (ITI-ALC) 
program’s  rapid  prototype  was  evaluated.  The  evaluation  focused  on  users’  perception  of 
how  well  the  prototype  met  system  and  human  computer  interface  requirements  for  PDM 
technicians  and  managers.  A  group  support  system  (GSS)  was  used  as  an  analysis  tool  to 
evaluate  the  prototype  and  collect  evaluation  data.  Using  the  prototype  as  a  requirements 
baseline  for  the  ITI-ALC  system,  this  thesis  had  three  objectives:  to  perform  an 
assessment  of  the  prototype  and  illicit  modifications;  to  determine  prototype 
compatibility  vwth  user’s  needs;  and  to  investigate  using  GSS  for  prototype  analysis.  A 
total  of  seven  users  composed  of  PDM  technicians  and  supervisors  evaluated  the 
prototype  by  following  a  scenario,  and  documenting  their  ideas  using  the  GSS.  Results 
indicate  the  prototype  functionally  meets  user’s  requirements,  however  suggested 
modifications  to  enhance  the  prototype  and  gain  more  user  acceptance.  Results  also 
indicate  that  a  GSS  is  effective  and  efficient  for  performing  prototype  analysis.  The 
primary  recommendation  was  to  make  suggested  changes  and  perform  further  tests  to 
refine  the  ITI-ALC  system  baseline. 
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ASSESSING  USER  REQUIREMENTS  FOR  AN  AUTOMATED  SYSTEM 
TO  SUPPORT  PROGRAMMED  DEPOT  MAINTENANCE 
THROUGH  USE  OF  A  RAPID  PROTOTYPE  IN  A 
GROUP  SUPPORT  SYSTEM  ENVIRONMENT 


T.  Tntroduction 

Background 

The  focus  of  this  study  will  be  to  assess  the  system  and  hiunan  computer  interface 
(HCI)  requirements  for  an  automated  information  system  to  support  aircraft  programmed 
depot  maintenance  (PDM).  The  mechanism  to  perform  this  task  is  a  rapid  prototype 
developed  for  the  Integrated  Technical  Information  for  the  Air  Logistics  Centers  (ITI- 
ALC),  a  research  and  development  program  being  developed  to  support  PDM  process. 
This  rapid  prototype  will  be  evaluated  by  qualified  depot  technicians  and  managers  to 
determine  how  well  it  meets  the  needs  of  the  intended  users  and  what  deficiencies 
currently  in  its  design. 

The  Air  Force  is  moving  toward  an  increasing  use  of  electronic  storage,  transfer, 
and  presentation  of  information  in  the  development,  support,  and  operation  of  weapon 
systems  (Masquelier,  1991).  The  force  behind  this  shift  to  using  digital  data  is  the 
Continuous  Acquisition  Life-Cycle  Support  (CALS)  initiative.  CALS  is  a  DoD  and 
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industry  initiative  enabling  the  integration  of  digital  technical  information  for  system 
acquisition,  design,  manufacture,  and  support  (Clark,  1992;  Chapman  &  Simmons,  1995). 
Currently,  much  of  the  research  and  development  toward  implementing  CALS  into  the 
support  environment  is  focused  on  the  delivery  of  digitized  technical  data  for  aircraft 
maintenance  technicians  working  on  the  flight  line. 

An  example  of  the  CALS  initiative  is  the  Integrated  Technical  Information  for  the 
Air  Logistic  Centers  (ITI-ALC).  ITI-ALC  is  a  developmental  program  that  is  extending 
information  integration  technology  into  the  Air  Logistic  Centers  for  the  support  of 
aircraft  depot  repair  and  inspection  (Masquelier,  1995).  The  objective  of  the  ITI-ALC 
program  is  to  analyze,  and  streamline  the  Programmed  Depot  Maintenance  (PDM) 
process  through  the  better  use  of  available  information.  Depot  maintenance  is 
responsible  for  all  the  scheduled  and  imscheduled  maintenance  of  aircraft  and  its 
associated  systems  and  components,  such  as  engines,  landing  gear,  and  avionics 
equipment.  An  effective  depot  maintenance  process  provides  operating  organizations 
with  sufficient  quantities  of  aircraft  and  serviceable  items  to  train  pilots  and  aircrews 
during  peace  and  to  fly  combat  missions  in  the  event  of  war.  Recent  improvements  in 
system  reliability  are  reducing  the  time  required  for  an  item  to  flow  through  the  depot 
process.  However,  this  benefit  is  constrained  by  decreasing  budgets  for  new  systems, 
weapon  system  spares,  and  for  training  technicians.  The  need  to  use  more  effective  and 
efficient  ways  to  carry  out  the  depot  maintenance  process  is  more  critical  today  than  ever 
before. 
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Computer  and  information  technology  advances  are  driving  many  projects  to 
improve  access  to  information  that  already  exists  within  maintenance  organizations  (AL, 
1995b).  Other  development  projects  have  been  directed  to  improve  tools  and  maintenance 
aids  for  technicians.  ITI-ALC  is  the  first  attempt  to  integrate  electronically,  the  available 
information,  tools,  and  aids  for  the  programmed  depot  maintenance  technician.  The  ITI- 
ALC  system  focuses  on  the  maintenance  technician’s  needs  as  the  most  important  aspect 
of  this  integration  process.  The  ultimate  value  of  the  ITI-ALC  system  and  its  acceptance 
by  the  end  user  is  linked  to  the  program’s  ability  to  achieve  measurable  performance 
improvements  at  the  technician  level  (Masquelier,  1996). 

ITI-ALC  encompasses  the  computer  and  information  technology  required  to  meet 
the  following  specific  objectives  (AL,  1995b:  1-1); 

1 .  integrate  multiple  maintenance  information  sources  into 
a  single,  easy-to-use  information  system 

2.  tailor  information  to  meet  the  specific  needs  of  the 
mechanic 

3 .  eliminate  time-consuming  paperwork  and  tasks  through 
automation 

4.  improve  the  quality  of  maintenance  performance  by 
taking  advantage  of  the  computer’s  ability  to  interact 
with  and  support  the  mechanic 

5.  maximize  available  manpower  resources  by  providing 
information  in  standard,  generic  formats  independent  of 
the  information  system,  and  by  supporting  general 
technical  capabilities  at  various  skill  levels 

6.  provide  a  link  to  the  organizational  level  of 
maintenance  to  implement  a  more  effective  transfer  of 
information  between  the  two  levels. 
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Problem  Statement 


A  rapid  prototype  has  been  developed  by  the  Logistics  Research  Division  of 
Armstrong  Laboratory  to  gather  the  information  needed  to  create  and  refine  a  software 
baseline  for  the  development  of  the  ITI-ALC  system  (Figure  1). 
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Figure  1.  Example  of  ITI-ALC  Prototype  Screen  (Quill,  1995) 

The  laboratory  is  interested  in  assessing  user  requirements  for  an  automated 
system  to  support  programmed  depot  maintenance.  As  a  means  to  evaluate  the  user 
requirements,  the  ITI-ALC  rapid  prototype,  that  is  based  on  previous  requirement 
analysis,  will  be  evaluated  in  a  Group  Support  System  (GSS)  environment.  The  purpose 
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of  the  evaluation  will  be  to  identify  any  deficiencies  in  the  current  design  relating  to 
missing  user  requirements  and  potential  user  interface  improvements.  Incomplete 
requirement  identification  can  lead  to  premature  implementation  of  a  system  that  does  not 
fully  meet  its  users  needs.  Likewise,  those  positive  aspects  that  stand  out  in  presenting 
the  information  will  also  be  identified  to  ensure  that  these  aspects  are  duplicated  or  at 
least  not  eliminated  in  the  design  of  the  ITI-ALC  demonstration  system.  Even  if  all  the 
requirements  are  implemented,  if  the  user  interface  is  developed  poorly  the  system  will 
not  be  accepted  or  used  by  the  maintenance  community. 

Research  Objectives 

Objectives  of  this  thesis  research  are  to:  (1)  perform  an  assessment  of  the  ITI-ALC 
rapid  prototype  and  elicit  suggestions  for  changes  or  modifications,  to  the  current  system 
requirements  that  are  necessary  for  the  initial  baseline  of  the  ITI-ALC  system,  (2) 
determine  its  compatibility  with  the  user’s  needs,  and  (3)  investigate  the  use  of  group 
support  systems  for  prototype  analysis. 

The  specific  investigative  questions  for  requirements  deficiencies  include: 

1 .  To  what  extent  has  the  current  prototype  adequately  captured  the  user 
requirements  for  managing  programmed  depot  maintenance  and  for  performing  a 
depot  maintenance  task? 

2.  If  the  prototype  does  not  meet  the  users  needs,  what  changes  or  modifications 
need  to  be  made  to  correct  the  prototype  system  so  that  it  does? 

3 .  Does  the  ITI-ALC  system  present  a  “Value-added”  to  the  user? 

4.  Does  using  a  group  support  system  aid  in  prototype  analysis? 
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The  specific  questions  for  the  usability  of  the  user  interface  will  include  (Shneiderman, 
1992): 

1 .  Does  the  prototype  meet  user  expectations  when  operating  the  system  (e.g.,  is 
it  consistent  throughout)? 

2.  Can  the  user  navigate  through  the  interface  easily? 

3.  Are  screens  organized  in  a  logical  manner? 

4.  Is  the  terminology  used  in  the  prototype  appropriate? 

5.  Does  the  prototype  minimize  potential  errors? 

6.  Is  maintenance  learning  enhanced  by  the  interface  and  prototype  learning 
minimized  (i.e.,  is  it  easy  to  learn)? 

7.  Are  the  system  capabilities  appropriate  (e.g.,  is  processing  time  too  slow)? 

Scope 

This  thesis  research  will  evaluate  the  ITI-ALC  rapid  prototype  developed  by 
Armstrong  Laboratory.  The  intent  is  to  identify  the  extent  and  nature  of  any  deficiencies 
presently  existing,  and  suggest  changes  or  modifications  that  should  be  considered  when 
developing  the  ITI-ALC  demonstration  system.  This  thesis  will  also  identify  any 
currently  missing  system  requirements  needed  to  establish  the  initial  baseline  of  the  ITI- 
ALC  system.  This  thesis  will  correspondingly  investigate  the  ITI-ALC  rapid  prototype’s 
compatibility  with  the  user’s  needs.  In  addition  to  evaluating  ITI-ALC,  this  thesis  will 
also  investigate  the  use  of  GSS  for  supporting  prototype  analysis. 
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II.  Literature  Review 


Introduction 

This  chapter  is  a  literature  review  of  the  areas  critical  to  the  success  of  this  thesis. 
It  was  conducted  to  gain  a  better  understanding  of  previous  work  and  research  that  has 
been  conducted  in  the  respective  areas. 

This  chapter  is  divided  into  four  main  sections.  Each  addresses  a  certain  aspect  of 
this  thesis  that  must  be  investigated  and  then  integrated  to  achieve  the  goals  set  for  this 
study.  The  first  section  is  a  high  level  look  at  the  Integrated  Technical  Information  for 
the  Air  Logistics  Centers  (ITI-ALC).  It  investigates  the  purpose,  objectives,  and  the 
overall  concept  of  the  program.  The  second  section,  reviews  details  of  the  rapid 
prototype  developed  to  refine  the  functional  and  human-computer  interface  of  the  ITI- 
ALC  system.  The  third  section  identifies  current  methods  for  performing  usability 
analysis.  The  fourth  section  tvill  investigate  group  support  systems  and  their  potential 
use  in  the  ITI-ALC  usability  analysis. 

Integrated  Technical  Information  for  the  Air  Logistics  Centers  (ITI-ALC) 

The  Integrated  Technical  Information  for  the  Air  Logistics  Centers  (ITI-ALC)  is  a 
research  program  being  conduct  by  the  Logistics  Research  Division  of  Armstrong 
Laboratory.  The  program  is  currently  in  its  early  stages  and  the  initial  requirements 
analysis  for  the  program  is  nearing  completion. 

The  purpose  of  ITI-ALC  is  to,  “primarily  improve  the  effectiveness  of  the 
mechanic  who  supports  the  weapon  system  at  the  depof’  (AL,  1995c:l).  This  can  be 
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achieved  by  providing  the  mechanic  access  to  all  the  information  necessary  to  perform 
the  aircraft  maintenance  without  them  having  to  leave  the  job  site.  For  example,  any 
technical,  diagnostic,  or  historical  information  that  the  mechanic  needs  to  perform 
maintenance  can  be  presented  through  a  single  device  using  single  interface.  ITI-ALC 
will  provide  access  to  integrated  planning,  scheduling,  aircraft  technical  orders,  and 
historical  information  to  name  a  few.  The  type  of  information  expected  to  be  linked  is 
graphically  depicted  in  figure  2  below. 


Figure  2.  ITI-ALC  Information  Integration  (AL,  1995) 

The  information  is  electronically  linked,  which  eliminates  the  need  for  the  mechanic  to 
physically  stop  work  and  retrieve  a  specific  Techniceil  Order  (TO)  that  he  does  not  have. 
The  fact  that  the  information  may  come  firom  several  different  data  bases  is  transparent  to 
the  user  because  all  the  information  is  presented  using  a  common  interface  and  no 
reference  to  the  source  of  the  information  is  made. 
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ITI-ALC  objectives 


In  order  to  make  the  goal  of  ITI-ALC  come  to  fruition,  six  specific  objectives 
were  identified  for  the  program.  The  first  was  to  integrate  all  the  maintenance 
information  that  the  mechanic  needs  into  a  single  information  system  that  is  easy  to  use. 
The  second  was  to  tailor  the  information  to  specifically  meet  the  needs  of  the  mechanic. 
The  third  was  to  automate  the  information,  thereby  eliminating  time  consuming  tasks  and 
paperwork.  The  fourth  was  to  improve  the  quality  of  maintenance  performance  by  taking 
advantage  of  the  human  computer  interaction  to  support  the  mechanic.  The  fifth  was  to 
maximize  the  available  work  force  by  providing  the  information  in  a  standard  generic 
format  that  is  independent  from  the  source  information  system.  This  generic  format 
would  be  able  to  support  varying  technical  levels  at  the  appropriate  skill  level.  The  sixth 
objective  was  to  maintain  a  link  to  the  maintenance  organizational  level  bases  and  create 
a  more  effective  information  transfer  between  the  depot  and  the  organizational  level. 

A  conceptual  system  configuration  needed  to  achieve  these  objectives  is 
represented  in  figure  3. 


9 


The  ITI-ALC  Concept 


Figure  3.  ITI-ALC  Conceptual  System  Configration  (AL,  1995c) 

In  figure  3,  the  databases  that  store  the  different  types  of  information  are  located  at  the  far 
left.  As  you  proceed  to  the  right,  the  multiple  display  devices  are  depicted  which  act  as 
the  means  transfer  the  information  from  the  databases  to  the  mechanic.  Going  to  the  far 
right,  the  data  is  sent  by  radio  frequency  (RF)  to  the  mechanic  at  the  aircraft. 

ITI-ALC  Requirement  Analysis  Approach 

To  meet  the  objectives  established  for  ITI-ALC,  the  cmrent  Programmed  Depot 
Maintenance  (PDM)  process  had  to  be  identified  and  analyzed  in  order  to  streamline  the 
process.  “The  process  improvement  was  initiated  from  an  information  perspective”  (AL, 
1995b:  1-2).  As  the  starting  point,  the  existing  depot  maintenance  process  was  used  to 
identify  the  initial  functional  requirements  and  to  derive  specifications  for  the  ITI-ALC 
system. 
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Systems  Research  and  Applications  Corporation  (SRA),  the  contracting  agency 
performing  the  requirements  analysis  for  Armstrong  Laboratory,  used  a  structured 
analysis  approach  in  building  the  information  models  and  functional  requirements  or  the 
ITI-ALC  system.  The  structured  analysis  approach  allows  the  user’s  view  of  the 
maintenance  environment  to  define  the  requirements  of  the  system.  “Successive 
decomposition  of  the  models  permits  different  groups  of  end  users,  to  view  the  system 
from  their  own  perspective”  (AL,  1995b:  1-2).  Within  context  of  the  problem,  each  user 
viewpoint  is  maintained,  while  simultaneously  providing  each  user  with  a  view  of  the 
system  that  is  familiar  to  them  all. 

Two  types  of  models  were  developed  during  this  requirements  analysis,  “As-Is” 
and  “To-Be.”  “As-Is”  models  depict  the  current  depot  maintenance  process  and  flow  of 
information.  The  “To-Be”  models  describe  an  improved  depot  maintenance  process. 
Analysis  of  the  “As-Is”  process  and  information  models  resulted  in  15  Business  Process 
Improvement  (BPI)  recommendations  (AL,  1995a).  The  BPIs  represent  the  addition  of 
state  of  the  art  technology,  changes  to  the  parts  acquisition  process,  and  depot  policy 
changes. 

The  “To-Be”  models  reflect  an  improved  and  streamlined  PDM  process  that  is 
expected  to  result  in  fewer  maintenance  flow  days  and  lower  operating  costs.  The  overall 
objective  of  the  ITI-ALC  program  is  to  improve  the  current  PDM  operations  and  to 
reduce  the  process  time  or  the  steps  needed  while  creating  a  more  effective  and  efficient 
PDM  line.  Emphasis  has  been  placed  on  early  integration  of  the  BPI’s  into  the  design  of 
the  ITI-ALC  system  in  order  to  reap  potential  cost  savings  upon  implementation. 
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The  accuracy  of  the  “As-Is”  and  “To-Be”  models  that  implement  BPI’s  are  critical 
to  the  success  of  the  ITI-ALC  program.  This  accuracy  can  only  be  achieved  by 
maintaining  a  constant  user  involvement  in  virtually  every  step  of  the  development 
process. 

User  Involvement 

In  similar  programs  proceeding  ITI-ALC,  the  Logistics  Research  Division  placed 
emphasis  on  the  continued  involvement  of  the  end  user.  This  philosophy  led  to  many 
program  successes.  The  same  philosophy  is  being  continued  in  the  ITI-ALC  program. 
The  requirement  analysis  (Phase  I)  of  the  program  began  with  interviews  of  those 
involved  with  every  aspect  of  the  aircraft  PDM  process.  The  interviews  identified  the 
functional  requirements  needed  for  the  ITI-ALC  system.  These  requirements  were 
documented  using  an  Integrated  Definition  Methodology  (IDEF)  which  was  used  to 
create  the  “As-Is”  and  the  “To-Be”  models.  The  models  were  then  taken  back  to  the 
users  at  the  depot  to  verily  the  process  and  information  flow.  This  iterative  process  for 
developing  the  models  continued  until  Armstrong  Laboratory  and  the  users  at  the  depot 
were  confident  of  the  models  accuracy. 

User  involvement  in  the  development  of  a  system  such  as  ITI-ALC  helps  produce 
a  better  product  and  increase  the  likelihood  of  the  users  acceptance.  It  also  gives 
developers  a  more  realistic  feel  for  the  amount  of  effort  needed  to  implement  the  user 
requested  features  (Boehm  et  al.,  1984).  The  next  step  for  the  program  was  to  develop 
the  rapid  prototype  that  would  initially  transform  the  requirements  defined  by  the  users 
into  a  testable  hiunan  computer  interface. 
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ITI-ALC  Rapid  Prototype 

The  ITI-ALC  rapid  prototype  demonstrates  the  concept  of  complete  information 
integration,  by  providing  through  a  single  seamless  interface,  access  to  plaiming, 
scheduling,  parts  ordering  and  maintenance  repair  information.  The  ITI-ALC  rapid 
prototype  was  developed  addressing  two  future  users:  the  PDM  mechanic  and  the  Aircraft 
Manager  (Quill,  1995).  Functional  flows  for  a  typical  PDM  process  were  developed 
using  maintenance  experts  from  Armstrong  Laboratory  and  depot  persoimel  from  Warner 
Robins  AFB.  Maintenance  experts  are  a  critical  step  in  the  building  process  because  they 
posses  an  in-depth  understanding  of  every  step  in  the  total  process.  These  functional 
flows  were  developed  by  integrating  information  from  the  IDEF  models  and  interviewing 
mechanics  and  aircraft  managers  at  the  depot.  The  functional  flows  were  used  to  create 
storyboards  for  the  rapid  prototype  and  to  show  the  BPI  recommendations.  The  rapid 
prototype  that  resulted  depicted  9  of  the  15  BPFs  planned  for  the  ITI-ALC  system  (Quill, 
1996). 

The  storyboards  for  the  rapid  prototype  were  not  complicated.  The  screens  were 
“simply  sketched  with  a  pencil  on  paper”  (Pfauth,  Hammer,  &  Fissel,  1985:467).  The 
screens  were  then  placed  in  a  logical  order  and  validated  with  the  maintenance  expert 
making  corrections  as  necessary.  The  result  was  a  typical  PDM  process  flow  on  paper. 

The  storyboards  that  created  a  “Paper  scenario”  were  then  prototyped  on  line 
using  Microsoft  Visual  Basic  and  Access.  The  result  was  a  rapid  prototype  that  has  the 
plaimed  HCI  features  and  functional  requirements  of  the  ITI-ALC  system. 
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The  purpose  of  the  ITI-ALC  rapid  prototype  is  two  fold.  First,  the  prototype  is 
used  as  a  demonstration  tool  to  help  visualize  the  ITI-ALC  concept.  Second,  it  is  used  as 
a  tool  to  obtain  feedback  from  expected  users  on  functional  and  the  user  interface 
requirements  (Weiser,  1982).  The  prototype  gives  a  good  representation  of  what  the  user 
can  expect  to  see  in  a  fully  operational  depot  system. 

There  are  many  benefits  in  using  prototyping  to  develop  a  user  interface. 
Prototypes  can  help  the  users  to  redefine,  evaluate,  and  analyze  their  needs  and 
requirements  very  early  in  the  design  process  (Finger  et  al.,  1996).  The  user  involvement 
in  the  development  process  also  helps  increase  the  likelihood  of  user  acceptance  of  the 
design  (Kroenke  &  Hatch,  1994).  In  addition,  prototyping  used  in  the  early  phases  of  a 
development  cycle  can  save  time  and  money  in  the  final  software  product  (Greitzer, 
1993).  According  to  Lange  and  Bhavnani  (1995:26),  “the  earlier  the  rapid  prototyping 
process  is  considered  in  a  program,  the  greater  the  rewards  are.” 

Major  benefits  to  prototyping  in  general  as  identified  by  Alavi  (1984)  and  Foley 
(1986)  and  restated  by  Wilson  and  Rosenberg  (1988:861)  are: 

(1)  Provides  a  means  for  testing  product  specific  questions  that  cannot  be 
answered  by  generic  research  and  guidelines. 

(2)  Provides  a  tangible  means  of  evaluating  a  given  user  interface  concept. 

(3)  Provides  a  common  reference  point  for  all  members  of  the  design  team,  users, 
and  marketing. 

(4)  Allows  the  solicitation  of  meaningful  feedback  from  users. 

(5)  Improves  the  quality  and  completeness  of  a  product’s  functional  specification. 

(6)  Increases  the  probability  that  the  product  will  perform  as  expected 

(7)  Substantially  reduces  the  total  development  cost  for  a  product. 


In  order  to  evaluate  the  rapid  prototype  correctly  and  obtain  the  benefits  outlined 
above,  current  testing  methods  and  techniques  were  reviewed  for  applicability.  The  focus 
for  the  testing  is  the  usability  of  the  rapid  prototype.  Since  the  rapid  prototype  possesses 
many  of  the  functional  and  HCI  requirements  of  a  folly  functional  system,  its  usability  is 
representative  of  the  folly  functional  ITI-ALC  system. 

Limitations  of  the  rapid  prototype 

The  ITI-ALC  rapid  prototype  used  in  this  study  currently  has  limited  robustness. 

It  also  has  limited  informational  links  to  Technical  Order  (TO)  and  job  guide  information. 
The  ITI-ALC  rapid  prototype  has  no  true  external  computer  interfaces  and  represents 
only  part  of  the  planning  and  scheduling  functional  requirements  envisioned  for  the  foil 
capability.  The  rapid  protot5q)e  does  however,  posses  sufficient  capability  to  answer  the 
investigative  questions  posed  for  this  thesis. 

Usability  testing 

Usability  is  a  term  used  through  this  thesis.  In  the  context  of  this  study,  usability 
means  that,  “the  people  who  use  a  product  can  do  so  quickly  and  easily  to  accomplish 
their  own  tasks”  (Dumas,  1993:4).  Dumas  further  elaborated  by  adding  that  the 
definition  rests  on  four  points  (Dumas,  1993:4): 

1.  Usability  means  focusing  on  users. 

2.  People  use  products  to  be  productive. 

3.  Users  are  busy  people  trying  to  accomplish  tasks. 

4.  Users  decide  when  a  product  is  easy  to  use. 

Since  it  is  virtually  impossible  to  anticipate  all  the  problems  that  can  be 
encountered  in  developing  a  system,  usability  testing  is  routinely  employed  as  a  means  to 
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identify  user  problems  early  in  the  design  phase.  A  multitude  of  variations  exists  in 
where  and  how  a  usability  test  should  be  conducted.  Common  to  all,  are  five 
characteristics  (Dumas,  1993:22): 

1 .  The  primary  goal  is  to  improve  the  usability  of  a 
product.  For  each  test,  you  also  have  more  specific  goals 
and  concerns  that  you  articulate  when  planning  the  test. 

2.  The  participants  represent  real  users. 

3.  The  participants  do  real  tasks. 

4.  You  observe  and  record  what  participants  do  and  say. 

5.  You  analyze  the  data,  diagnose  the  real  problems,  and  recommend  changes  to 
fix  those  problems. 

Usability  testing  typically  involves  the  use  of  task  scenarios.  A  task  scenario  is  a 
descriptive  story  describing  typical  use  of  the  system.  It  tells  the  user  contextually,  what 
they  are  supposed  to  do  during  the  test,  but  refrains  from  providing  detailed  specifications 
of  the  task  steps.  A  realistic  scenario  also  helps  to  take  some  of  the  artificiality  out  of  the 
test. 


The  task  scenario  approach  as  a  tool  in  usability  testing  is  well  accepted  and  was 
adopted  in  this  study.  The  evaluation  method  needed  to  evaluate  the  rapid  prototype  was 
not  as  clear,  so  a  further  review  of  traditional  evaluation  methods  was  conducted. 
Usability  Evaluations 

According  to  Hewitt  (1986)  and  Booth  (1992),  usability  evaluations  can  be 
divided  into  two  separate  types  depending  on  the  kind  of  information  needed  for  the 
evaluation.  The  two  types  are  formative  and  summative.  A  formative  evaluation  usually 
requires  qualitative  information  that  helps  the  designer  refine  a  design  and  identify  those 
parts  of  the  system  that  need  changing.  Summative  information,  conversely,  is  more 
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likely  to  require  quantitative  information.  Summative  evaluation  involves  assessing  the 
overall  performance  of  the  user  and  the  system  (Booth,  1992). 

Different  evaluations  are  used  at  different  stages  in  the  design  of  a  system.  For 
instance,  qualitative  information  is  needed  when  refining  a  design.  It  provides  locations 
of  errors  and  more  explanation  about  the  problems  that  the  users  encounter.  In  the  later 
stages  of  design,  the  quantitative  information  becomes  more  valuable.  The  quantitative 
information  helps  assess  the  usefulness  of  the  changes  that  have  been  made  in  the  system. 

Once  the  type  of  information  needed  for  the  usability  test  is  determined,  the 
method  for  obtaining  the  information  can  be  chosen.  According  to  Nielsen  and  Mack 
(1994),  there  are  four  basic  ways  of  evaluating  the  usability  of  an  interface: 
automatically,  empirically,  formally,  and  informally.  An  automatic  evaluation  is  one 
where  the  usability  measures  are  computed  by  running  the  specification  for  the  user 
interface  through  an  evaluation  software  to  obtain  results.  An  empirical  evaluation  is  one 
where  the  usability  of  a  product  is  assessed  by  testing  the  interface  with  actual  users  of 
the  end  product.  The  formal  evaluation  involves  using  exact  models  and  formulas  to 
calculate  usability  measures.  Finally,  an  informal  evaluation  is  based  on  general  rule  of 
thumb  such  as,  “looking  and  feeling  right,”  and  the  general  skill,  knowledge  and 
experience  of  the  users  performing  the  evaluation. 

Armstrong  Laboratory  has  always  focused  on  having  the  actual  end  item  user 
provide  feedback  for  making  modifications  or  changes  to  a  demonstration  system 
(Johnson,  1991).  The  end  item  user  is  normally  in  the  best  position  to  know  what  is 
needed.  End  item  users  however,  may  have  differing  levels  of  computer  experience. 
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In  recent  years  there  has  been  a  substantial  growth  in  the  number  of  users  who  are 
not  computer  experts.  An  expert  computer  user  is  an  individual  that  works  with 
computers  frequent  and  is  comfortable  with  the  task.  To  illustrate,  a  naive  user’s 
computer  experience  may  be  limited  to  computer  games,  or  DOS  based  programs  used  on 
the  job  and  those  tasks  may  be  rather  repetitive.  Eason  (1976)  termed  these  users  as 
“naive.” 

The  methods  for  obtaining  usability  data  vary  depending  on  the  goal  of  the 
evaluation  and  the  users  opinion  that  is  being  solicited.  Different  users  will  look  at  the 
same  prototype  from  a  different  viewpoint  and  as  a  result  provide  different  kinds  of 
information  in  their  feedback.  By  specifying  the  objective  of  the  evaluation  it  is  possible 
to  identify  what  measures  you  are  seeking  and  what  method  should  be  used. 

When  it  can  be  assumed  that  the  user  is  naive,  standard  measurements  such  as 
time,  number  of  errors,  and  qualitative  information  identifying  what  caused  the  errors  can 
be  used  as  valid  measurement  in  the  evaluation  (Booth,  1992).  When  it  can  not  be 
assumed  that  all  the  users  participating  in  the  evaluation  are  naive,  the  method  used  may 
be  a  combination  of  existing  methods.  Many  methods  of  evaluation  are  available  besides 
testing  with  naive  users.  Some  of  these  methods  as  outlined  by  Booth  (1992: 124-127) 
are: 

Concept  test.  A  concept  test  is  where  only  the  theory  or  concepts  of  the  system  are  laid 
out  on  a  card  or  piece  of  paper  and  the  presented  to  the  user.  The  objective  of  this  test  is 
to  identify  those  concepts  that  are  acceptable,  and  those  that  are  not  that  can,  potentially 
create  confusion. 
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Friendly  user.  Testing  with  friendly  users  involves  using  s  from  other  groups  that  have 
interest  in  the  system.  Friendly  users  will  usually  have  enough  technical  knowledge  to 
suggest  changes  that  might  be  made  to  the  system.  Friendly  users  are  very  useful  in  that 
they  possess  a  reasonably  good  knowledge  of  the  system,  whereas  naive  users  do  not.  A 
negative  aspect  to  this  method  is  that  friendly  users  tend  to  miss  aspects  of  the  system 
that  cause  problems  for  the  naive  user  simply  because  the  naive  user  does  not  possess  that 
increased  system  knowledge. 

Hostile  user.  A  hostile  user  is  someone  who  has  no  vested  interest  in  the  system  and 
therefore  loses  nothing  if  the  system  fails.  The  hostile  user  may  identify  inconsistencies 
or  flaws  in  their  critical  review  that  would  have  created  problems  for  the  naive  user.  This 
is  to  say  that  a  hostile  user  is  more  likely  to  be  critical  than  a  friendly  user. 

Simulating  users.  Simulating  users  is  accomplished  by  charting  the  navigation  through 
the  system  of  a  naive  user.  Later,  when  the  time  is  more  appropriate,  that  same  system 
routing  is  replicated  by  designers.  This  approach  provides  an  advantage  in  that  the 
designer  may  take  routes  that  were  not  anticipated  in  the  original  system  design. 
Structured  Walkthrough.  This  approach  is  very  similar  to  the  simulating  users  approach. 
Designers  in  this  method,  walk  through  expected  task  routings  searching  for  potential 
user  pitfalls.  The  difference  between  the  structured  walkthrough  and  the  simulation  is 
that  in  a  structured  walkthrough  the  designers  take  paths  that  they  think  the  user  would 
use  and  simulation  is  based  on  actual  user  routing. 

Expert  review.  In  an  expert  review,  the  system  is  evaluated  by  a  human  factors  engineer 
or  designer  that  is  not  associated  with  the  project.  This  hopefully  removes  any  bias  for. 
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or  against  the  system.  This  approach  has  an  advantage  in  that  the  comments  are  coming 
from  someone  who  is  knowledgeable  about  human  computer  interfaces.  A  disadvantage 
to  this  approach  is  that  the  expert  is  also  doing  the  same  type  of  work,  even  though  not  on 
the  same  project,  as  the  system  designers.  This  creates  a  potential  for  them  to  also 
overlook  some  difficulties  that  the  naive  user  will  encounter. 

Simulation  trials.  Simulation  trials  involve  using  prototypes  and  mock-ups  to  test  the 
intended  system  design.  This  method  is  usually  used  before  a  large  portion  of  time  and 
effort  is  expended  on  the  first  iteration  of  the  design.  It  eliminates  the  designers  feeling 
of  extreme  ownership  and  reluctance  to  change.  This  method  also  contributes  to  saving 
time  and  money  because  changes  can  be  made  earlier  in  the  design. 

Iterative  informal  laboratory  experiments.  This  approach  focuses  on  iterative  refining  of 
prototypes  (Booth,  1992).  In  this  approach,  qualitative  comments  are  used  to  iteratively 
modify  a  prototype,  thereby  eliminating  as  many  problem  areas  as  possible. 

Formal  laboratory  experiments.  Formal  laboratory  experiments  add  the  element  of 
control.  This  approach  reduces  the  likelihood  of  the  experimenter  effecting  the 
evaluation.  Quantitative  data  are  usually  obtained  and  used  to  either  support  or  refute  an 
established  hypothesis  in  this  method.  Disadvantages  are  that  the  users  are  removed  from 
the  normal  working  environment.  This  creates  doubts  that  the  responses  of  the  user  are  a 
reliable  indicator  of  the  system  being  tested.  Moreover,  this  approach  tends  to  capture  the 
users  initial  response  to  the  system  and  learning  issues,  rather  than  the  ease  of  use  and 
efficiency  of  the  system  (Booth,  1992). 
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Audits.  Audits  are  when  the  systems  are  is  evaluated  using  checklists  created  from 
human  factors  literature.  These  checklists  may  include  areas  such  as  screen  layout,  or 
previously  established  hardware,  or  software  requirements. 

Field  trials.  Field  trials  are  when  the  system  is  put  into  the  organization  that  it  was 
developed  for,  prior  to  formal  release.  This  allows  realistic  studying  of  the  system,  but 
can  be  time  consuming  and  make  it  difficult  to  gather  data. 

Follow-up  studies.  This  approach  is  used  to  refine  a  system  that  has  already  been  put  into 
operation.  Changes  to  the  system  from  follow-up  studies  usually  come  in  the  form  of 
version  updates. 

Field  studies.  Field  studies  are  conducted  within  the  organizations  where  the  system  is 
being  used.  They  are  t5q)ically  more  extensive  than  either  field  trials  or  follow-up  studies 
and  are  harder  to  control  than  a  formal  experiment.  The  major  advantage  to  this  method 
is  that  it  provides  an  accurate  representation  of  how  much  the  system  is  actually  being 
used. 

This  list,  although  not  exhaustive,  provides  an  idea  of  some  of  the  methods 
available  to  perform  evaluations.  “In  reality,  these  methods  are  seldom  used  in  a  pure 
form,  but  evaluation  tends  to  be  something  of  a  mix  and  match  as  far  as  methods  are 
concerned  (Booth,  1992:127).” 

The  evaluation  conducted  in  this  thesis  will  be  a  formative  evaluation  in  that  it 
requires  qualitative  information  that  will  be  used  by  designers  to  refine  the  ITI-ALC  rapid 
prototype  and  identify  areas  that  do  not  meet  the  users  needs.  The  evaluation  is  also 
empirical  according  to  the  definition  used  by  Nielson  (1994),  because  the  product  will  be 
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assessed  with  actual  users  of  the  end  item  product.  The  method  that  will  be  used  in  this 
study  is  not  as  clear.  Using  Booth’s  list  of  methods  for  using  non-naive  users,  the  method 
appears  to  be  a  combination  of  friendly  user,  simulation  trial,  and  one  iteration  of  an 
iterative  informal  laboratory  experiment.  To  go  a  step  further  in  combining  methodology 
types,  this  thesis  will  explore  using  a  Group  Support  System  (GSS)  to  capture  the  ideas 
generated  by  the  users  for  refining  the  rapid  prototype. 

Using  a  Group  Support  System 

Over  the  past  decade  researchers,  have  explored  a  range  of  issues  regarding  the 
use  of  networked  computers  to  support  multiple  users  or  teams  in  decision  making 
(Heminger,  1988).  GSS  have  been  developed  to  help  groups  share  their  ideas  or  thoughts 
on  a  matter,  organize  these  thoughts,  and  then  vote  on  the  options  leading  to  a  group 
consensus  on  an  action.  “GSS  has  shown  to  increase  the  effectiveness  and  efficiency  of 
teams,  by  supporting  and  enhancing  beneficial  group  processes,  while  circumventing  or 
minimizing  many  of  the  counter  productive  aspects  of  group  work”  (Heminger  et  al., 
1994:2).  No  instances  however,  could  be  foimd  where  the  GSS  has  been  used  in 
conducting  a  usability  analysis  of  any  kind. 

The  GSS  appears  to  be  an  imtapped  resource  that  could  provide  a  new  way  of 
conducting  usability  analysis.  At  the  very  least,  it  presents  an  extreme  potential  for  a 
faster  and  more  efficient  way  of  documenting  qualitative  information  during  a  usability 
test.  It  also  provides  a  means  of  allowing  multiple  users  to  simultaneously  evaluate  a 
product.  A  third  benefit  of  using  a  GSS  in  usability  testing  is  that  individual  evaluations 
can  be  conducted  and  the  results  immediately  transferred  into  a  group  environment.  The 
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ideas  generated  by  the  individual  are  still  “fresh”  in  the  their  minds  and  the  group  can 
now  discuss  the  ideas  and  possibly  generate  even  more  ideas  as  a  result  of  the  group 
discussion. 

Group  Support  Systems 

GSS  is  a  computer  based  system,  composed  of  a  facility,  hardware,  software, 
procedures  and  facilitation  working  together  to  support  and  augment  the  efforts  of  work 
groups  to  complete  an  unstructured  task  (Heminger  et.  al.,  1994:2).  To  date,  GSS’s  have 
been  primarily  used  to  support  business  teams  and  quality  action  groups.  This  is 
expected  to  expand  as  the  benefits  of  GSS  are  realized.  Usability  analysis  may  be  such  a 
case. 

The  benefits  of  GSS  can  be  related  to  problems  encountered  in  many  traditional 
meetings  (Fraase,  1991).  For  example,  time  can  be  wasted  as  people  “go  off  on  tangents” 
and  stray  from  the  issue  at  hand.  Perhaps  the  personalities  of  a  few  of  the  participants 
may  tend  to  dominate  the  meeting  making  the  results  biased  toward  those  few 
individuals.  In  many  cases,  ideas  can  be  lost  because  some  users  are  shy  or  reserved  or 
perhaps  fear  reprisal.  Lastly,  the  minutes  or  records  of  the  meeting  are  subjective  or 
incomplete,  and  in  some  cases  contradictory.  This  problem  can  be  compounded  when 
researchers  rely  on  memory  to  recount  the  events  because  they  have  become  “task 
saturated”  as  multiple  deficiencies  are  being  quickly  discovered.  The  same  problems 
could  be  experienced  during  a  group  usability  analysis.  These  problems  can  cause 
meetings  to  become  inefficient,  ineffective,  and  unproductive. 
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Three  characteristics  of  a  GSS  that  make  the  meetings  more  productive  can  also 
make  a  usability  analysis  more  productive.  They  are:  anonymity,  simultaneous  and 
parallel  processing  of  the  information,  and  full  and  immediate  recording  of  the 
information.  The  anonymity  of  the  GSS  system  allows  users  to  enter  their  comments  and 
ideas  without  the  other  users  knowing  the  input  source.  Those  users  that  typically  are 
quiet  and  more  reserved  in  traditional  meetings,  are  able  to  be  more  creative  and  assertive 
in  entering  their  ideas.  This  allows  ideas  to  be  presented,  that  in  a  traditional  meeting 
might  not  surface.  The  simultaneous  and  parallel  processing  allows  the  users  to  enter 
their  ideas  at  the  same  time.  Each  user  has  the  potential  to  "feed  off "  the  others  ideas 
allowing  group  ideas  to  more  quickly  evolve.  This  characteristic  can  sdso  serve  to  speed 
up  the  analysis  and  accelerate  what  can  be  accomplished  because  everyone  is  presenting 
their  ideas  at  once,  yet  all  the  ideas  are  being  presented  without  interruption.  Finally,  the 
full  and  immediate  recording  of  the  information  and  transpiring  ideas  are  automatically 
recorded  and  available  for  printing  immediately  after  the  meeting.  The  researchers  do  not 
have  to  rely  memory  or  the  accuracy  of  the  individual  recording  the  results  of  the 
analysis. 

Summary 

This  chapter  investigated  four  areas  critical  to  the  success  of  this  thesis:  the 
Integrated  Technical  Information  for  the  Air  Logistics  Centers  (ITI-ALC)  program,  the 
ITI-ALC  rapid  prototype  developed  by  Armstrong  Laboratory,  different  methods  for 
performing  usability  analysis,  and  group  support  systems. 
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The  Integrated  Technical  Information  for  the  Air  Logistics  Centers  (ITI-ALC) 
program  was  researched  in  order  to  understand  the  concept  of  the  system  and  what  is 
required  by  its  expected  user.  Continuous  user  involvement  has  been  the  standard  for 
developing  the  ITI-ALC  system.  Literature  indicates  that  this  type  of  philosophy 
produces  a  better  system  at  a  lower  cost  and  fosters  more  of  a  “buy  in”  support  for  the 
system  from  the  end  item  user  (Greitzer,  1993). 

The  second  area  researched  in  this  literature  review  was  the  ITI-ALC  rapid 
prototype  developed  by  Armstrong  Laboratory.  The  prototype  is  a  next  step  in  the 
iterative  design  process.  It  builds  on  and  implements  the  initial  requirements  analysis  for 
the  program.  It  also  provides  a  platform  to  help  verify  functional  and  human  computer 
interface  requirements  for  a  fully  functional  information  system.  The  prototype 
highlights  business  process  improvements  that  are  expected  to  save  money  and  time  over 
the  current  PDM  process. 

In  order  to  accurately  evaluate  the  rapid  prototype,  several  methods  for  evaluating 
the  usability  of  a  system  were  reviewed.  The  method  chosen  is  a  combination  of  three 
standard  methods:  Friendly  user.  Simulation  trials,  and  Iterative  informal  laboratory 
experiments.  These  three  methods  will  be  combined  to  tailor  a  method  for  this  study. 
The  resulting  method  will  be  coupled  with  an  apparently  rmtested  approach  of  using  a 
group  support  system  to  perform  the  usability  analysis. 

Group  support  systems  (GSS)  were  investigated  as  a  possible  imtapped  resource 
that  can  be  used  in  prototype  analysis.  GSS  may  be  a  means  to  more  effectively  and 
efficiently  perform  usability  analysis  given  their  successes  in  group  processes.  The 
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anonymity  it  provides  to  users  combined  with  the  automatic  documentation  of  ideas 
makes  it  an  intriguing  possibility. 

This  literature  review  is  the  starting  point  for  this  research  that  is  to  perform  an 
evaluation  of  a  rapid  prototype  that  has  been  developed  for  the  Integrated  Technical 
Information  for  the  Air  Logistics  Centers.  The  following  chapter  will  describe  the 
methodology  used  to  evaluate  the  ITI-ALC  rapid  prototype. 
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III.  Methodology 


Chapter  Overview 

This  chapter  explains  the  methodology  necessary  to  evaluate  the  ITI-ALC  rapid 
prototype.  First  this  chapter  will  present  the  experimental  design  and  the  projected  study 
results.  Second,  it  will  describe  the  equipment  used  during  the  evaluation.  It  will  then 
discuss  the  tasks  and  the  subjects  chosen  for  the  study  and  finally,  it  will  explain  the  data 
collection  procedures  and  analysis  necessary  to  answer  the  objectives  outlined  for  this 
thesis. 

Throughout  this  thesis,  the  participants  in  this  study  will  be  referred  to  as  users. 

In  addition,  the  ITI-ALC  rapid  prototype  will  be  referred  to  as  the  prototype,  and  the 
group  support  system  used  to  collect  data  will  be  referred  to  as  the  GSS. 

Experimental  Design 

Objectives 

The  experimental  design  for  this  study  was  developed  to  achieve  3  major  objectives. 
The  objectives  are  to:  (1)  perform  an  assessment  of  the  ITI-ALC  rapid  prototype  and 
elicit  suggestions  for  changes  or  modifications,  to  the  current  system  requirements  that 
are  necessary  for  the  initial  baseline  of  the  ITI-ALC  system,  (2)  determine  the  ITI-ALC 
rapid  prototype’s  compatibility  with  the  user’s  needs,  and  (3)  investigate  the  use  of  group 
support  systems  for  prototype  analysis. 
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Method 

The  method  of  evaluation  defined  for  this  design  was  a  combination  of  three 
methods  outlined  by  Booth  (1992).  The  design  was  a  simulation  trial,  conducted  with  a 
single  iteration  laboratory  experiment  using  friendly  users.  Users  utilized  the  rapid 
prototype  during  an  evaluation,  and  made  comments  as  they  went  through  the  depot 
process  scenario.  Because  of  their  experience,  the  users  provided  an  element  of  system 
knowledge,  about  the  process  that  the  ITI-ALC  demonstration  system  is  being  designed 
for. 

Sample  population 

The  target  group  for  this  study  was  aircraft  depot  maintenance  technicians, 

functional  supervisors  or  managers  for  the  aircraft  programmed  depot  maintenance  line, 

at  Warner  Robins  ALC,  GA.  All  users  sampled  were  either  currently  performing  the  job 

they  represent,  or  have  recent  experience  in  that  position.  Seven  users  were  chosen  from 

the  target  group  to  evaluate  the  ITI-ALC  rapid  prototype. 

Seven  users  have  been  deemed  adequate  for  this  study  based  on  a  finding 
by  Robert  Virzi  that  80  percent  of  usability  problems  can  be  detected  with 
four  to  five  subjects  and  the  addition  of  more  subjects  is  less  likely  to 
reveal  any  more  problems  in  the  application  being  evaluated.  Virzi  went 
on  to  say  that  the  most  severe  problems  in  the  application  tend  to  be 
detected  by  the  first  few  subjects.  (Virzi,  1992) 

Projected  Study  Results 

The  expected  results  of  this  study  and  their  anticipated  use  by  the  sponsoring 
agency  are  as  follows: 

1.  Improved  understanding  of  ITI-ALC  suitability  in  satisfying  user  needs  by 
identifying  those  rapid  prototype  features  that  worked  well,  and  those  did  not. 
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2.  Feedback  from  the  users  in  this  study  will  provide  insight  into  the  changes  that 
should  be  made  to  the  evolving  design  concepts  for  the  ITI-ALC  system  development 
and  those  features  which  currently  meet  or  exceed  user’s  needs. 

3.  The  ideas  collected  in  this  study  will  be  made  available  to  Armstrong 
laboratory  to  provide  to  the  contractor  developing  the  ITI-ALC  demonstration  system. 

4.  This  study  will  illustrate  the  potential  value  of  group  support  systems  in  future 
usability  studies. 

Hardware 

The  ITI-ALC  rapid  prototype  and  group  support  system  were  presented  on  386- 
based  computers  operating  at  25  MHz  with  16  MB  of  RAM.  All  computers  are  in  the 
groupware  laboratory  located  in  room  319  of  building  641,  at  the  Air  Force  Institute  of 
Technology  (AFIT).  These  computers  and  terminals  are  typical  of  those  expected  to  be 
used  to  support  the  fully  functional  ITI-ALC  demonstration  system  in  the  air  logistics 
centers. 

Software 

The  software  for  this  study  includes  two  separate  software  packages  running  with  a 
Windows  3.1  operating  system.  The  first  is  the  ITI-ALC  rapid  prototype  developed  by 
Armstrong  Laboratory.  This  rapid  prototype  was  developed  using  Microsoft  Visual 
Basic  3.0  Professional  and  Microsoft  Access  2.0.  The  second  software  package  is  the 
GSS  and  is  Ventana  GroupSystem  for  Windows.  This  package  will  serve  as  the 
collaborative  software  to  facilitate  data  collection  for  the  rapid  prototype  evaluation. 
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Each  user  was  asked  to  review  the  rapid  prototype  and  document  the  results  of  the 
evaluation  using  the  group  support  system.  The  capability  to  switch  between  active  tasks 
was  granted  by  using  the  Alt-tab  key  combination  available  in  Microsoft  Windows.  This 
capability  allowed  the  user  to  review  the  rapid  prototype  and  immediately  switch  to  the 
GSS  to  document  their  ideas  or  comments  about  the  rapid  prototype. 

Room  Equipment 

The  room  was  equipped  with  8  user  terminals  and  1  facilitator  terminal  coimected  in 
a  LAN  network.  A  user  terminal  is  a  terminal  where  the  person  that  is  performing  the 
evaluation  sits.  This  terminal  is  granted  control  by  the  facilitator  terminal.  The  facilitator 
terminal  is  the  master  terminal  where  the  GSS  software  is  controlled.  The  terminals  were 
arranged  on  tables  placed  on  tables  pushed  together  at  the  sides  because  of  the  space 
limitation  in  the  room.  It  also  had  a  single  overhead  projector  equipped  with  LCD  panel 
display  to  allow  the  information  being  presented  on  the  facilitator  terminal  to  be 
simultaneously  viewed  by  the  group  on  an  overhead  screen.  This  allows  the  group  to 
look  at  the  same  screen  and  conduct  group  discussions.  The  room  was  also  equipped 
with  two  white  boards  to  aid  in  group  discussions. 

Task 

During  the  evaluation,  each  user  reviewed  the  ITI-ALC  rapid  prototype  by  stepping 
through  an  established  scenario  (Appendix  B).  As  the  users  performed  this  walkthrough 
of  the  established  scenario,  they  documented  their  comments  using  the  group  system 
software.  The  group  system  software  used  for  this  study  was  the  Ventana  GroupSystem 


30 


for  Windows.  This  software  served  as  a  tool  to  collect  and  present  the  ideas  for 
improving  or  modifying  the  rapid  prototype.  These  ideas  were  subsequently  rated  on  a 
scale  from  1  to  10  by  each  individual.  The  rating  results  will  then  combined,  creating  an 
initial  ranking  of  all  the  group  ideas 

The  user  centered  workshop  plan  in  Appendix  A,  describes  the  details  of  the 
evaluation.  It  details  the  sequence  of  events  to  include  introductions,  tutorials  and  the 
evaluation  phases. 

During  the  evaluation,  observations  were  made  of  the  users  interacting  with  the 
prototype.  These  observations  were  documented  on  packages  that  showed  each  prototype 
screen  in  the  scenario  in  the  same  sequence  as  the  users  were  looking  at  during  the 
evaluation. 

After  the  evaluation,  each  user  was  asked  to  fill  out  a  questionnaire  that  addressed 
specific  areas  dealing  with  the  usability  of  the  prototype.  The  areas  of  this  usability 
questionnaire  correspond  with  areas  typically  looked  at  by  HCI  engineers  when 
developing  an  information  system. 

Data  Collection 

The  data  collected  in  this  study  is  largely  qualitative  in  nature.  Two  data 
collection  tools  were  used,  questionnaires  and  the  group  support  system.  Two  separate 
questionnaires  were  given  to  each  user.  The  first  was  a  biographical  questiormaire  that 
was  seeking  personal  information,  depot  experience,  and  the  individuals  familiarization 
with  computers  and  software  packages  (Appendix  C).  The  second  questiormaire  was 
directed  toward  the  usability  of  the  prototype  (Appendix  D).  This  questiormaire  was 
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administered  at  the  conclusion  of  the  test  to  gather  any  remaining  comments  or 
suggestions  as  well  as  their  overall  impression  of  the  prototype. 

Ideas  and  comments  obtained  during  the  evaluation  of  the  prototype  contribute  to 
the  information  needed  to  answer  the  functional  and  user  interface  questions  identified  in 
chapter  1 .  The  rating  of  the  user’s  ideas  was  also  collected  using  the  group  support 
software  and  established  a  prioritization  identifying  how  important  each  particular 
requirement  is  to  the  user  (Nielson,  1992). 

Observations  in  the  form  of  notes  and  comments  on  a  paper  copy  of  the  scenario 
were  also  collected  during  the  session.  (Appendix  H).  The  notes  served  to  capture 
information  subtleties  that  constitute  the  intangible  nuances  or  “feel”  of  the  user  using  the 
prototype. 

Controls 

The  goal  of  this  study  was  to  gather  uninhibited  ideas  on  the  ITI-ALC  rapid 
prototype  to  identify  missing  requirements  and  usability  problems  of  the  system  prior  to 
its  final  design.  The  nature  of  this  study  was  to  therefore  have  limited  controls  and  to 
place  minimal  restrictions  on  the  group.  Some  controls  however,  were  unavoidable  in 
order  to  maintain  the  integrity  of  the  results.  These  controls  are  as  follows: 

1 .  All  users  were  from  one  base  to  eliminate  variability  in  accepted  local 
maintenance  operating  procedures. 

2.  The  group  walked  through  a  single  scenario  during  the  initial  evaluation  to  ensure 
that  every  user  saw  the  same  screens  and  was  asked  to  perform  the  same  tasks. 
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3.  The  Group  software  laboratory  was  isolated  from  outside  activities  to  keep 
distractions  to  a  minimum. 

Data  Analysis 

The  following  information  measurements  will  be  used  to  analyze  and  draw 
conclusions  toward  answering  the  objective  questions  identified  in  chapter  1  of  this 
thesis. 

a.  A  ranked  list  of  ideas  identifying  potential  changes  to  the  rapid  prototype. 

b.  The  consolidated  ideas  will  be  categorized  into  groups  or  areas  of  concern. 

c.  Explanatory  comments  provided  by  the  users  for  each  idea  will  help  clarify  the 
meaning  and  provide  recommendations  on  missing  human-computer  interface  or 
functional  requirements. 

d.  Individual  assessment  of  the  prototype  as  obtained  in  the  usability  questionnaire. 
Specific  investigative  questions  and  the  means  for  answering  the  questions  to 

identify  requirement  deficiencies  are  as  follows: 

1 .  To  what  extent  has  the  current  prototype  adequately  captured  the  user 
requirements  for  managing  programmed  depot  maintenance  and  for  performing  a  depot 
maintenance  task? 

Measurements  1, 2,  and  3  in  combination  with  measurement  4,  the  usability  questionnaire 
items  1-26  contributed  to  answering  this  investigative  question. 

2.  If  the  prototype  does  not  meet  the  user’s  needs,  what  changes  or  modifications 
need  to  be  made  to  correct  the  prototype  system  so  that  it  does? 
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Measurements  1, 2,  and  3  in  combination  with  measurement  4,  the  usability  questionnaire 
items  1-26  contributed  to  answering  this  investigative  question. 

Measurements  1, 2,  and  3  specifically  address  the  information  needed  to  answer  this 
investigative  question. 

3.  Does  the  ITI-ALC  system  present  a  “Value-added”  to  the  user? 

Item  questions  1, 2,  and  24  of  the  usability  questionnaire  provide  the  information  needed 
to  answer  this  investigative  question. 

4.  Does  using  a  group  support  system  aid  in  prototype  analysis? 

Only  three  questions  on  the  usability  questioimaire  addresses  this  question.  Item  27, 28, 
and  29  seek  the  feedback  that  answers  this  question. 

The  specific  questions  on  usability  include  (Shneiderman,  1992); 

1 .  Does  the  prototype  meet  user  expectations  when  operating  the  system  (e.g.,  is 
it  consistent  throughout)? 

Items  1, 2,  and  3  of  the  usability  questionnaire  seek  the  information  for  this  user  interface 
investigative  question. 

2.  Can  the  user  navigate  through  the  interface  easily? 

This  question  is  addressed  by  various  portions  of  measurement  1, 2,  and  3  but  is  not 
solely  from  any  one.  Items  6-10  of  the  usability  questionnaire  specifically  address  this 
investigative  question. 

3.  Are  screens  organized  in  a  logical  manner? 

Measurements  1, 2,  and  3  address  aspects  of  this  question  along  with  items  4  and  5  of  the 
usability  questionnaire. 
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4.  Is  the  terminology  used  in  the  prototype  appropriate? 

As  with  many  of  the  previous  investigative  questions,  measurement  1, 2,  and  3  in  part 
address  this  question.  Items  1 1-13  of  the  usability  questionnaire  directly  addresses  this 
question. 

5.  Does  the  prototype  minimize  potential  errors? 

Measurement  4,  the  usability  questionnaire  directly  addresses  this  question  in  items  14- 
17. 

6.  Is  maintenance  learning  enhanced  by  the  interface  and  prototype  learning 
minimized  (i.e.,  is  it  easy  to  learn)?  Items  18-21  in  the  usability  questionnaire  addresses 
this  investigative  question. 

7.  Are  the  system  capabilities  appropriate  (e.g.,  is  processing  time  too  slow)? 
Items  22  and  23  in  the  usability  questionnaire  addresses  this  investigative  question. 
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IV.  Results  and  Analysis 


Chapter  Overview 

This  study  was  designed  to  gather  information  in  three  forms:  group  prototype 
evaluation  results,  individual  prototype  usability  results,  and  observations  made  on  user- 
prototype  interaction.  This  chapter  describes  the  subjects  used  in  the  evaluation,  presents 
the  results  obtained  in  each  form  of  data  collection  and  analyzes  the  result  as  they  pertain 
to  each  of  the  investigative  questions  stated  for  this  thesis. 

Subjects 

The  evaluation  of  the  rapid  prototype  was  conducted  using  a  total  of  7  users 
composed  of  programmed  depot  maintenance  (PDM)  technicians  and  functional 
supervisors  or  managers  for  the  aircraft  depot  line  at  Warner  Robins  ALC.  Each  user 
evaluated  the  prototype  based  on  personal  knowledge  of  the  depot  maintenance  process, 
the  information  currently  needed  to  perform  the  maintenance  and  planning  tasks,  and  the 
manner  that  the  information  was  presented  to  them.  User  testing  such  as  this  can  answer 
many  questions  that  cannot  be  addressed  using  a  generic  user  (Gruenenfelder  and 
Whitten,  1984).  The  individuals  personal  knowledge  and  expertise  in  the  depot 
maintenance  process  were  invaluable  for  an  accurate  review  of  the  functional 
requirements  and  the  graphical  user  interface  used  to  present  the  technical  orders  and 
information  to  the  user  of  the  ITI-ALC  demonstration  system  (Plato,  1995). 
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Prototype  Evaluation  Results 

The  results  collected  in  each  of  the  methods  in  many  cases  provide  overlapping  or 
duplicate  information.  Conversely,  some  information  that  was  collected  was  found  to  be 
unique  to  the  method  used  in  the  data  collection. 

GSS  Results 

The  prototype  evaluation  using  the  GSS  was  a  3  step  process.  The  first  step  was 
to  generate  ideas  and  supporting  comments  as  the  users  walked  through  a  pre-defined 
scenario.  The  second  step  was  to  rate  each  idea  according  to  the  idea’s  criticality  to  a 
fully  functional  ITI-ALC  system  as  perceived  by  the  users.  The  third  step  was  to  place 
the  ideas  into  categories  typically  used  by  human  factors  engineers  in  Armstrong 
Laboratory  when  evaluating  information  systems  and  their  user  interfaces.  The 
categories  were  user  expectation,  screen,  navigation,  terminology,  errors,  learning,  and 
rapid  prototype  capabilities. 

Evaluation  step  1 : 

The  GSS,  in  the  first  step  of  the  evaluation,  captured  user  ideas  and  comments 
relating  to  the  prototype.  Those  ideas  and  comments  are  listed  in  table  1 . 

Table  1 .  User  generated  ideas  and  comment  for  the  ITI-ALC  prototype  using  a  GSS 

GSS  Documented  Momiation 

IDEA  GENERATED  BY  THE  USERS  _  ^  ^  - 

•  comments  made  to  support  and  explain  the  idea  generated _ 

1.  TECHNICAL  ORDER  GRAPfflCS  WERE  INSUFFICIENT  IN  SOME  AREAS 

•  graphic  doesn’t  show  gap  tolerance _ _ 

•  picture  of  rack  not  sufficient _ 

•  graphics  should  be  displayed  full  view  on  entry  and  be  able  to  zoom  in  on  specific 
areas 
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Table  1.  User  generated  ideas  and  comments  for  the  ITI-ALC  prototype  using  a  GSS 

(Continued) 


•  need  more  than  one  view  on  locator  area  around  work  needs  to  be  more  detailed 

•  add  menu  capability  for  exploded  views. 


2.  TECHNICAL  ORDER  PRESENTATIONS  WERE  INSUFFICIENT  IN  SOME 
AREAS 


•  need  capability  to  mark  last  completed  step  (e.g.,  bookmark) _ 

•  warnings  before  starting  tasks  are  very  helpful. _ 

•  to  ensure  proper  recognition  of  icons  computer  classes  for  all  mechanics  a  must. 


3.  PROBLEMS  WITH  REVIEWING  AIRCRAFT  PREVIOUS  WRITE-UPS 

•  needs  to  be  reworded  to  reflect  "previous"  in  lieu  of  o  level _ 

•  bad  menu  choice  o-level  history _ 

•  spell  out  "o  level"  to  organization _ 

•  don’t  abb,  menu  items _ 

•  have  write  ups  come  up  before  job  is  started _ 

•  how  do  you  get  to  previous  write-ups 


4.  CHANGE  DISPLAY  OF  AIRCRAFT  OPERATION  STATUS _ 

•  status  bar  is  difficult  to  read,  would  rather  see  a  time  line  of  operation  showing 
completion 


5.  MODIFY  CHARACTERISTICS  OF  AIRCRAFT  IDENTIFICATION _ 

•  Selection  of  aircraft  should  be  consistent  with  other  selections.  Single  mouse  click 

without  watch  symbol  confusing.  Double-clicking  caused  work  form  to  update. _ 

•  are  these  aircraft  tail  numbers?  which  is  assigned  to  me?  highlighted  aircraft  signify 

managers  aircraft _ 

•  identify  which  aircraft  belongs  to  which  aircraft  manager,  this  information  would  be 

useful  in  can  items,  (read  only) _ 

•  provide  history  of  aircraft  configuration 

-  which  TCTO’s  have  been  complied  with? 


_ -  what  special  work  has  been  performed? 

•  need  bldg  location  i.e.  bldg  number  and  bay 


6.  MASTER  SCHEDULE  ..BAR  GRAPH  COLORS  NEED  TO  BE  EXPLAINED. 
EX.  RED-  BEHIND  SCHEDULE,  (Legend) 


7.  GET  READY  TO  DO  MAINTENANCE 

•  on  parts  source  what  is  none _ 

•  need  to  reflect  between  age  and  test  equipment 
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Table  1 .  User  generated  ideas  and  comments  for  the  ITI-ALC  prototype  using  a  GSS 

(Continued) 


•  need  to  show  ordering  status 

•  can't  perform  task  unless  equipment  is  available 

•  need  further  instruction  on  how  to  order  equipment 

8.  PLANNING  INTERFACE  REQUIREMENT  NEEDED 

•  need  interface  with  planning  organization  as  with  problems  in  process  such  as  TCTO 
compliance,  e.g.,  cams  indicates  a  TCTO  being  c/w  when  actually  it's  not 

•  need  interface  with  engineering  for  202  requests  and  also  AFTO  22  requests 

9.  MECHANIC  SHOULD  BE  ABLE  TO  RETURN  TO  ANY  TECHNICAL 

ORDER  TASK  WHEN  DOCUMENTING  MAINTENANCE  PERFORMED 

10  PERTINENT  AIRCRAFT  WRITE  UPS  WITH  WORK  AREAS 

•  Group  tasks  by  physical  location. 

11.  173  CARDS  NEED  TO  BE  SHOWN  ON  THE  SCREEN  WITH  SPECIFIC 
NUMBER  OF  PERSONNEL  REQUIRED 

•  inspection  code  should  be  predetermined,  not  an  option.  Critical  requires  2,  non- 
critical  requires  1 

•  Need  to  allow  one  signature  box  for  each  person  signing  the  "card." 

12.  ADD  FIELDS  IN  SUPPORT  EQUIPMENT  SCREEN 

•  need  to  show  part  number 

•  fed.  stock  number,  and  nomenclature. 

13.  CAPABILITY  TO  REINITIATE  TECHNICAL  ORDER  ALERTS  WHILE 
PERFORMING  MAINTENANCE 

•  have  capability  to  reinitiate  alert  from  the  menu  or  icon 
(step  7). 

•  like  aircraft  locator 

14.  SCREEN  SHOWING  AGE  EQUIPMENT  BACKORDERED  WAS  UNCLEAR 

•  has  equipment  on  backorder  already  been  ordered  or  does  mechanic  need  to  order? 

15.  HOW  TO  CLOSING  OUT  WAS  NOT  CLEAR  FROM  TASK  TO  TASK 

•  wording  unclear  could  not  find  sign  off 

•  have  an  exit  button  to  log  out 

16.  HOW  TO  MAKE  A  MECHANIC  ASSIGNMENT  WAS  CONFUSING 

•  did  not  know  clicking  on  mechanic  name  was  way  of  selecting  name 

39 


Table  i.  User  generated  ideas  and  comments  for  the  ITI-ALC  prototype  using  a  GSS 

(Continued) 


17.  THE  LOGON  GRAPHICS  WERE  GREAT! 

•  Aircraft  fly-by _ 


18.  PROCESS  WAS  EXCELLENT 

•  scenario  was  excellent 


Based  on  group  discussions,  the  ideas  are  further  described  as  follows: 

Idea  1.  “Technical  order  graphics  were  insufficient  in  some  areas.”  This  is  in 
reference  to  graphics  within  the  digital  technical  orders  (T.O.).  The  graphics  presented 
are  directly  from  the  paper  T.O.’s  presently  used  by  depot  technicians.  The  comments 
made  by  the  users'  center  around  the  capability  to  see  the  details  of  the  graphic  within  the 
window  provided  in  the  prototype.  One  other  comment  highlights  the  omission  of  a  gap 
tolerance  that  was  either  already  presents  in  the  T.O.,  or  omitted  in  the  scanning  and 
“cleaning”  process  of  digitizing  the  T.O.  data. 

Idea  2.  “Technical  order  presentations  were  insufficient  in  some  areas.”  This  is  a 
very  global  statement.  One  comment  refers  more  to  learning  the  standard  toolbar 
configuration  or  presentation  than  it  does  the  actual  TO  presentation.  Another 
recommends  ITI-ALC  provide  a  “bookmark”  capability  to  enable  the  user  to  mark  a  T.O. 
step  and  return  to  it  later. 

Idea  3.  “Problems  with  reviewing  aircraft  previous  write-ups."  This  idea  centers 
on  using  the  menu  selection  for  “0-level  history”  on  the  aircraft.  The  menu  items  were 
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not  intuitive  to  the  user.  Another  comment  suggested  to  eliminate  the  need  to  go  to  the 
menu  for  the  history  because  of  the  confusion. 

Idea  4.  “Change  display  of  aircraft  operation  status.”  This  idea  suggests  a  single 
line  that  is  appropriately  filled  as  work  is  completed  on  the  aircraft  be  used  instead  of  the 
current  method.  This  proposed  method  is  currently  implemented  by  users  and  manually 
tracked  on  a  scheduling  board. 

Idea  5.  “Modify  characteristics  of  aircraft  identification.”  This  idea  refers  to  the 
depot  floor  screen  that  allows  the  aircraft  manager  to  select  aircraft.  One  comment 
suggests  that  the  prototype  be  consistent  with  the  number  of  mouse  clicks  needed  to 
activate  the  selection.  The  remaining  comments  suggest  capabilities  or  information  that 
should  be  added  to  that  screen. 

Idea  6.  “Master  schedule.. bar  graph  colors  need  to  be  explained.  Ex.  red-behind 
schedule.”  The  users  agreed  that  a  legend  be  implemented  to  identify  what  the  colors 
mean  on  the  master  schedule. 

Idea  7.  “Get  ready  to  do  maintenance.”  This  covers  a  functional  term  referring  to 
the  process  of  making  sure  the  user  has  the  parts  and  tools  needed  to  perform  a 
maintenance  task.  These  items  are  located  under  the  main  menu  item  “Prepare  for 
maintenance.”  The  comments  listed  under  this  idea  deal  with  fields  in  screens  below  sub¬ 
menu  item  “supp  equip  status”  and  “kit  status.” 

Idea  8.  “Planning  interface  requirement  needed.”  This  idea  suggests  interfaces  be 
included  for  planning  purposes  that  currently  do  not  exist.  The  comments  on  this  idea  are 
very  well  explained  in  Table  1. 
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Idea  9.  “Mechanic  should  be  able  to  return  to  any  technical  order  task  when 
documenting  maintenance  performed.”  Currently  the  user  is  unable  to  go  back  to  a  T.O. 
step  once  he  has  completed  the  maintenance  task  and  began  to  document  the  action.  This 
may  be  necessary  in  order  to  accurately  document  the  task  and  is  not  currently  available. 

Idea  10.  “Group  pertinent  aircraft  write-ups  with  work  areas.”  This  idea  is 
suggesting  to  sort  the  write-ups  for  an  aircraft,  so  that  only  those  of  concern  to  a  specific 
work  area  are  presented.  This  eliminates  the  need  to  search  through  all  the  write-ups  to 
find  those  that  are  applicable  to  a  specific  work  center. 

Idea  II.  “173  cards  need  to  be  shown  on  the  screen  with  specific  number  of 
personnel  required.”  This  idea  suggests  that  the  number  of  people  performing  the 
inspection  documented  on  the  173  card  should  not  be  a  choice,  but  rather  predetermined. 
Tasks  that  require  critical  inspection  codes  need  2  inspectors'  signatures,  while  non- 
critical  inspection  coded  tasks  require  only  one  inspector  signature. 

Idea  12.  “Add  fields  in  support  equipment  screen.”  This  idea  suggests  adding 
fields  for  part  number,  federal  stock  number  and  nomenclature  to  the  support  equipment 
status  screen  that  is  located  under  menu  item  “Prepare  for  Maintenance.” 

Idea  13.  “Capability  to  reinitiate  technical  order  alerts  while  performing 
maintenance.”  The  user  is  suggesting  they  be  able  to  reinitiate  an  active  alert  from  both 
the  menu  and  the  icon. 

Idea  14.  “Screen  showing  age  equipment  backordered  was  unclear.”  The  context 
of  what  the  backordered  statement  was  stating  can  be  misimderstood.  Terminology 
should  be  clearer. 
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Idea  15.  “How  to  close  out  was  not  clear  from  task  to  task.”  The  file  menu 
fimction  to  log  out  of  the  system  was  not  clear  to  the  user.  Users  suggested  that  a 
separate  button  be  created  to  make  logging  out  faster  and  easier. 

Idea  16.  “How  to  make  a  mechanic  assignment  was  confusing.”  The  user  was 
unaware  that  double  clicking  on  the  mechanic’s  name  would  initiate  the  assignment 
screen. 

Idea  1 7.  “The  logon  graphics  were  great! .”  The  bit  map  of  the  aircraft  flying 
across  the  screen  was  appealing  to  the  users. 

Idea  18.  “Process  was  excellent.”  The  user  appeared  to  think  that  the  scenario 
that  they  used  when  evaluating  the  prototype  was  an  accurate  depiction  of  the  depot 
process. 

Evaluation  step  2: 

In  step  2  of  the  evaluation,  each  user  rated  the  importance  of  the  ideas  using  a  10 
point  scale.  The  ideas  most  critical  to  the  ITI-ALC  system  were  rated  a  10  and  the  least 
critical  ideas  were  rated  at  1 .  Table  1  ideas  are  listed  as  ordered  by  the  users.  The  most 
critical  idea  is  listed  at  the  beginning  table  and  the  least  critical  is  listed  at  the  end. 
Details  such  as  number  of  users  selecting  a  specific  rating,  the  group  mean  and  group 
standard  deviations  for  each  idea  are  shown  in  Appendix  E. 

Evaluation  step  3: 

In  step  three  of  the  evaluation,  users  placed  the  rated  ideas  into  categories.  The 
predetermined  categories  were  user  expectation,  screen,  navigation,  terminology,  errors, 
learning,  system  capability,  and  miscellaneous.  The  purpose  of  placing  the  ideas  into 
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categories  was  to  relate  each  idea  back  to  the  questionnaire  and  the  usability 
measurements.  Unfortimately,  this  did  not  work  well.  The  ideas  generated  by  the  users 
were  functionally  oriented  as  opposed  to  human  factors  oriented  toward  presentation  or 
style  issues.  In  this  case,  the  ideas  were  directed  toward  performing  maintenance.  This 
made  it  difficult  for  the  users  to  place  the  ideas  into  human  factors  categories  simply 
because  they  were  not  as  familiar  with  them.  To  compound  the  problem,  the  categorizing 
was  performed  at  the  end  of  the  work  day  and  the  users  were  eager  to  complete  the  study 
and  retire.  As  a  result,  the  categories  that  the  users  placed  the  ideas  in  did  not  provide 
valuable  information  for  supporting  ITI-ALC  design  activities  and  were  eliminated  from 
this  chapter.  The  information  from  the  original  categorizing  process  is  contained  in 
Appendix  F.  It  shows  all  ideas  and  supporting  comments  placed  within  the  defined 
categories  as  determined  by  the  users. 

Usability  Ouestionaire  Results 

The  second  instrument  used  to  collect  data  was  the  usability  questionnaire.  The 
questions  and  results  are  separated  into  three  different  tables.  Separating  the  information 
should  to  make  it  easier  to  read  and  identify  pertinent  information.  The  10  point  scale 
questions  are  in  Table  2.  The  numbers  to  the  right  of  the  questions  indicate  the  number 
of  users  that  made  a  particular  response.  It  should  be  noted  that  the  median  for  this  scale 
used  in  this  questionnaire  is  5.5.  Responses  above  the  median  are  considered  positive 
responses  and  those  below  are  considered  negative  responses.  The  open  end  questions 
are  located  in  Table  3  and  the  5  point  agree/disagree  questions  are  in  Table  4. 
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Each  question  and  the  results  for  that  question  are  presented  immediately 
following  the  questionnaire  tables.  Histograms  for  the  response  to  the  scaled  questions  as 
well  as  the  original  comments  the  open  end  questions  are  in  Appendix  G. 

The  histograms  provide  a  very  good  graphical  presentation  of  the  user  responses.  They 
also  provide  a  better  picture  of  the  magnitude  and  spread  of  the  responses  made  by  the 
users. 
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Table  3.  Usability  questionnaire  open  ended  questions 


Usability  Questionnaire 

Open-end  questions 

Questions 

Responses 

24.  What  did  you  particularly  like  about  ITI- 
ALC? 

1 .  My  grand  kids  will  love  it 

2.  Working  process 

3.  It  gives  basically  all  information 
that  is  needed  to  accomplish  a  work 
task. 

4.  Something  to  help  mech. 

5.  Overall  sequence  was  great, 
flowing  from  one  step  to  another, 
fairly  easy  with  minor  problems. 

6.  Flow  of  work  process,  easier  to 
track  work  and  parts 

7.  Step  by  procedure 

25.  What  in  particular  did  you  NOT  like 
about  ITI-ALC? 

1 . 1  will  be  dead  before  I  get  to  use  it 

2.  No  instructions  up  front  to 

navigate. 

3.  Nothing 

4.  Some  commands  were  a  little 
misleading,  some  steps  hard  to 
follow.  Not  enough  information  on 
views  of  work  areas. 

5.  Overall  system  needs  to  be  more 
user  friendly.  More  T.O.  information 
needs  to  be  installed. 

6.  Nothing 

26.  What  recommendations  would  you  make 
for  improving  ITI-ALC? 

1 .  Try  to  make  it  more  user  friendly 

2.  Need  more  schooling  on  computer 
systems 

3.  Make  it  more  user  friendly. 

4.  Have  a  few  more  studies 

5.  Provide  user  friendly  computer 
classes  to  ensure  continuity 
between  operators. 

6.  More  input  from  mechanics. 

7.  Have  a  presentation  on  basic 
computer  skills 
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Table  4.  Usability  questionnaire  5  point  scale  questions 


Usability  Questionnaire 

5  Point  scale  questions 

Questions 

Number  of  users  responding  to  scale  value 

Strongly 

Agree 

Agree 

Neutral 

Disagree 

Strongly 
i  Disa^ee. 

27. 1  would  use  a  Group  Support  System  to 
evaluating  another  prototype. 

5 

2 

28.  The  GSS  helped  me  to  evaluate  the 
rapid  prototype  faster  than  if  I  didn't  use 
it. 

4 

3 

29.  The  GSS  was  effective  in  helping  me 
evaluate  the  rapid  prototype. 

■  -4 

3 

HI 

m 

Question  1.  “Overall,  use  of  the  system  was:”  The  range  of  the  answers  was  from 
frustrating  (1)  to  satisfying  (10).  The  results  show  an  overall  positive  tendency  with  a 
cluster  around  the  mid-range.  Two  of  the  seven  users  assigned  ratings  below  5.5  that  is  a 
negative  rating. 

Question  2.  “Overall,  use  of  the  system  was.”  The  range  of  the  answers  was  from 
difficult  (1)  to  easy  (10).  The  results  show  an  overall  positive  tendency  with  a  cluster 
around  the  mid-range.  Four  users  assigned  a  6  rating  with  two  below  5.5. 

Question  3.  “The  needs  of  both  experienced  and  inexperienced  users  were 
considered:”  Possible  answers  ranged  from  “never”  (1)  to  “always”  (10).  Responses  to 
this  question  are  varied  with  no  real  tendency  being  apparent. 

Question  4.  “The  characters  on  the  computer  screen  were:”  The  answers  range 
from  “hard  to  read”  (1)  to  “easy  to  read”  (10).  The  responses  range  mostly  across  the 
positive  side  of  the  scale.  All  but  one  response  to  this  question  was  positive. 
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Question  5.  “The  sequence  of  information  the  screen  was:”  The  answers  range 
from  “disorganized”  (1)  to  “organized”  (10). .  The  results  show  an  overall  positive 
tendency  with  two  clusters.  The  first  is  at  the  high  range  and  the  second  is  at  the  mid¬ 
range.  Only  one  response  was  negative. 

Question  6.  “Logging  into  the  system  and  starting  the  task  was:”  Possible 
answers  ranged  from  “difficult”  (1)  to  “  easy”  (10). ).  The  results  show  an  overall 
positive  tendency  with  a  clustering  in  the  positive  range.  Only  one  response  was 
negative. 

Question  7.  “Navigation  among  item  (e.g.,  fill-ins)  on  the  screen  was:”  The 
answers  range  from  “illogical”  (1)  to  “logical”  (10).  The  results  show  all  responses  were 
positive  and  were  rated  either  6  or  7  on  the  scale. 

Question  8.  “Instructions  on  how  to  navigate  from  screen  to  screen  was:”  The 
answers  range  from  “not  helpful”  (1)  to  “helpful”  (10).  The  results  show  an  overall 
negative  tendency  with  a  cluster  around  the  mid-range.  One  user  rated  the  navigation  at  a 
1. 

Question  9.  “The  sequence  of  flow  from  one  screen  to  the  next  was:”  The 
answers  range  from  “illogical”  (1)  to  “logical”  (10).  The  results  show  an  overall  positive 
tendency  with  all  responses  being  7  or  8.  All  but  one  response  was  rated  7. 

Question  10.  “Logging  off  the  system  was:”  The  answers  range  from  “difficult” 
(1)  to  “easy”  (10).  Responses  showed  no  clustering.  Five  of  the  seven  responses  were 
positive  with  2  being  rated  in  the  negative  range. 
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Question  11.  “The  terminology  used  throughout  the  system  was:”  The  answers 
range  from  “hard  to  understand”  (1)  to  “easy  to  understand”  (10).  The  results  show  an 
overall  positive  tendency  with  a  cluster  around  the  mid-range.  One  rating  was  a  1 0  and 
one  fell  negative 

Question  12.  “In  relation  to  the  work  you  do,  the  use  of  terminology  was:”  The 
answers  range  from  “inconsistent”  (1)  to  “consistent”  (10).  The  results  show  an  overall 
positive  tendency  with  a  cluster  around  the  mid-range.  Only  one  response  is  in  the 
negative  range. 

Question  13.  “The  terminology  used  in  messages  was:”  The  answers  range  from 
“not  helpful”  (1)  to  “helpful”  (10).  The  results  show  an  overall  positive  tendency  with  a 
cluster  aroimd  the  mid-range.  Four  of  the  seven  users  rated  a  7  for  the  message 
terminology. 

Question  14.  “Correcting  mistakes  was:”  The  answers  range  from  “difficult”  (1) 
to  “easy”  (10).  The  results  show  a  split  between  positive  and  negative  ratings.  However, 
five  of  the  seven  users  indicated  a  positive  response  with  two  indicating  a  negative 
response. 

Question  15.  “The  number  of  ITI-ALC  system  failures  encoimtered  was:”  The 
results  show  a  clustering  at  the  lower  part  of  the  scale.  Note  that  for  this  question,  the 
response  is  positive. 

Question  16.  “Number  of  error  messages  encountered  was:”  The  results  vary 
across  the  scale  and  show  no  tendency. 
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Question  1 7.  “Recovering  from  errors  was:”  The  answers  range  from  “difficult” 
(1)  to  “easy”  (10).  The  results  vary  greatly  across  the  scale  ranging  from  1  to  7.  No 
tangible  results  can  be  seen. 

Question  18.  “Learning  how  to  use  the  system  was:”  The  answers  range  from 
“difficult”  (1)  to  “easy”  (10).  The  results  show  and  overall  positive  tendency.  Six  of  the 
seven  responses  are  positive. 

Question  19.  “Remembering  the  names  and  use  of  commands  was:”  The  answers 
range  from  “difficult”  (1)  to  “easy”  (10).  All  responses  are  positive.  One  response  is 
rated  a  6  with  the  remaining  evenly  split  among  7  and  8. 

Question  20.  “Exploration  of  system  functions  was:”  The  answers  range  from 
“rigid”  (1)  to  “flexible”  (10).  All  but  one  response  is  positive.  Those  positive  responses 
vary  relatively  evenly  across  the  positive  half  of  the  scale. 

Question  21.  “The  task  flow  between  the  real  worked  and  the  system  seemed  to  be 
the:”  The  answers  range  from  “different”  (1)  to  “same”  (10).  The  results  show  an  overall 
positive  tendency  with  two  responses  being  negative.  The  positive  responses  are  evenly 
distributed  from  7  to  10. 

Question  22.  “Response  time  from  the  system  was:”  The  answers  range  from 
“slow”  (1)  to  “fast”  (10).  All  responses  are  positive  and  rated  at  7,  8,  or  9. 

Question  23.  “Overall,  use  of  the  system  was:”  The  answers  range  from 
“unreliable”  (1)  to  “reliable”  (10).  All  responses  were  positive  with  five  responses  being 
rated  8. 
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Question  24.  “What  did  you  particularly  like  about  ITI-ALC?”  Two  comments 
identified  positive  reaction  to  the  system  providing  the  information  needed  by  the 
mechanic.  Four  responses  addressed  the  flow  of  information.  The  first  being  that  a 
system  such  as  ITI-ALC  would  make  it  easier  to  track  work  and  parts.  The  second 
indicated  the  flow  of  information  as  he  went  through  the  prototype  was  “fairly  easy  with 
minor  problems.  The  third  and  fourth  responses  dealing  with  the  flow  of  information 
were  unclear  to  exactly  what  they  were  trying  to  say.  The  last  users  comment  indicated 
that  “his  grand  kids  will  love  it.”  The  exact  meaning  to  this  response  could  take  several 
avenues,  but  is  taken  as  being  a  positive  reaction  to  the  prototype. 

Question  25.  “What  did  you  particularly  NOT  like  about  ITI-ALC?”  Three 
comments  dealt  with  the  user  interface.  The  first  comment  suggested  that  it  needed  to  be 
more  user  friendly.  The  second  comment  said  that  some  commands  were  a  little 
misleading.  The  third  indicated  that  some  T.O.  steps  were  hard  to  follow  with  not 
enough  information  being  provided  to  make  instructions  clear.  Two  more  responses  said 
there  was  nothing  they  disliked  about  the  ITI-ALC  prototype  and  the  last  suggested  that 
he  would  be  dead  before  he  would  get  to  use  it. 

Question  26.  “What  recommendations  would  you  make  for  improving  ITI-ALC?” 
This  question  was  asked  to  obtain  recommendations  firom  the  depot  user  on  what  they 
thought  would  improve  ITI-ALC  system.  Two  users  suggested  that  more  studies  should 
be  conducted  on  the  system  with  even  more  input  coming  from  the  mechanic.  Two  other 
responses  suggested  that  the  system  should  be  made  more  user  fiiendly,  but  made  no 
specific  suggestions  on  what  would  make  the  system  more  user  fiiendly.  The  three 
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remaining  user  comments  dealt  with  receiving  training  on  how  to  use  computers  and  the 
system. 

Question  27.  “I  would  use  a  Group  Support  System  to  evaluate  another 
prototype.”  All  responses  were  either  agree  or  strongly  agree. 

Question  28.  “The  Group  Support  System  helped  me  to  evaluate  the  rapid 
prototype  faster  than  if  I  didn’t  use  it.”  All  responses  were  either  agree  or  strongly  agree 
that  the.  prototype  was  faster  than  if  they  did  not  use  it. 

Question  29.  “The  Group  Support  System  was  effective  in  helping  me  evaluate 
the  rapid  prototype.”  Again,  all  responses  were  either  agree  or  strongly  agree. 

Observation  Results 

The  third  method  of  gathering  data  was  to  observe  the  users  and  write  down 
observations  during  the  study.  During  the  prototype  evaluation,  researchers  from 
Armstrong  Laboratory  assisted  in  monitoring  the  users  as  they  evaluated  the  prototype. 
The  researchers  were  both  familiar  with  the  ITI-ALC  prototype  and  well  versed  in 
gathering  user  feedback  for  developmental  information  systems.  The  compiled  notes 
identify  areas  where  the  users  were  having  trouble  on  a  given  screen.  Researchers  also 
assisted  the  users  in  “getting  back  on  track”  within  the  scenario  when  they  became 
excessively  confused  on  what  to  do  next.  The  notes  were  documented  in  packages 
showing  each  screen  in  the  same  sequence  as  the  user's  evaluation  scenario.  The 
individual  notes  were  later  consolidated  into  one  package  shown  in  Appendix  H.  The 
notes  indicate  63  observations  were  made  by  the  researchers  while  the  users  evaluated  the 
prototype. 
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Analysis 

The  study  provided  data  relevant  to  the  4  investigative  questions  that  were 
identified  in  chapter  1  of  this  thesis.  Each  will  be  addressed  in  turn  in  the  following 
paragraphs. 

Invesigative  question  1 

To  what  extent  has  the  current  prototype  adequately  captured  the  user  requirements  for 
managing  programmed  depot  maintenance  and  for  performing  a  depot  maintenance 
task? 

Investigative  question  1  required  the  use  of  both  instruments  and  the  observation 
notes.  The  first  instrument  was  the  GSS  gathered  ideas  and  comments.  The  second 
instrument  was  26  questions  fiom  the  usability  questionnaire.  The  third  form  of 
information  was  the  researcher  observation  notes. 

The  results  of  the  GSS  ideas  and  comments  are  shown  in  Table  1  of  the  GSS 
results.  The  questions  in  the  usability  questionnaire  that  contribute  information  to 
answering  this  investigative  question  are  1-26.  Histogram  for  each  question  is  in 
Appendix  G. 

User  expectations  of  the  prototype.  The  questions  pertaining  to  the  collection  of 
the  user  expectations  of  the  prototype  are  as  follows: 

Question  1.  “Overall,  use  of  the  system  was:”  The  results  show  an  overall 
positive  tendency  toward  satisfying,  with  a  cluster  around  the  mid-range. 
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Question  2  “Overall,  use  of  the  system  was.”  The  results  show  an  overall 
positive  tendency  toward  easy  to  use  with  a  cluster  around  the  mid-range  with  four 
responses  rated  6. 

Question  3.  “The  needs  of  both  experienced  and  inexperienced  user  was 
considered:”  Responses  varied  with  no  real  tendency  being  apparent. 

Screen  organization.  The  questions  pertaining  to  the  screen  organization  of  the 
prototype  are  as  follows: 

Question  4.  “The  characters  on  the  computer  screen  were:”  Responses  range 
mostly  across  the  positive  side  of  the  scale.  All  but  one  response  to  this  question  was 
positive. 

Question  5.  “The  sequence  of  information  the  screen  was:”  Results  show  an 
overall  positive  tendency  with  two  clusters.  The  first  is  at  the  high  range  and  the  second 
is  at  the  mid-range.  Only  one  response  was  negative. 

Navigation.  The  questions  pertaining  to  the  navigational  ease  of  the  prototype  are 
as  follows: 

Question  6.  “Logging  into  the  system  and  starting  the  task  was:”  Results  show 
an  overall  positive  tendency  with  a  clustering  in  the  positive  range.  Only  one  response 
was  negative. 

Question  7.  “Navigation  among  item  (e.g.,  fill-ins)  on  the  screen  was:”  Results 
show  all  responses  were  positive  and  were  rated  either  6  or  7  on  the  scale. 
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Question  8.  “Instructions  on  how  to  navigate  from  screen  to  screen  was:”  Results 
show  an  overall  negative  tendency  with  a  cluster  around  the  mid-range.  One  user  rated 
the  navigation  at  a  1. 

Question  9.  “The  sequence  of  flow  from  one  screen  to  the  next  was:”  Results 
show  an  overall  positive  tendency  with  all  responses  being  7  or  8.  All  but  one  response 
was  rated  7. 

Question  10.  “Logging  off  the  system  was:”  Five  responses  were  positive  to  this 
question. 

Terminology.  The  questions  pertaining  to  the  adequacy  of  the  terminology  used 
in  the  prototype  are  as  follows: 

Question  11.  “The  terminology  used  throughout  the  system  was:”  Results  show 
an  overall  positive  tendency  with  a  cluster  around  the  mid-range.  One  rating  was  a  10 
and  one  fell  negative 

Question  12.  “In  relation  to  the  work  you  do,  the  use  of  terminology  was:” 

Results  show  an  overall  positive  tendency  with  a  cluster  around  the  mid-range.  Only  one 
response  is  in  the  negative  range. 

Question  13.  “The  terminology  used  in  messages  was:”  Results  show  an  overall 
positive  tendency  with  a  cluster  around  the  mid-range.  Four  of  the  seven  users  rated  a  7 
for  the  message  terminology. 

Minimize  potential  error.  The  questions  pertaining  to  the  prototype’s  tendency  to 
minimize  potential  errors  when  using  it  are  as  follows: 


56 


Question  14.  “Correcting  mistakes  was:”  Results  show  an  unequal  split  between 
positive  and  negative  ratings.  Five  users  indicated  a  positive  response  with  two 
indicating  a  negative  response. 

Question  15.  “The  number  of  ITI-ALC  system  failures  encountered  was:”  The 
results  show  a  clustering  at  the  lower  part  of  the  scale.  Note  that  for  this  question,  this  is 
within  the  positive  range. 

Question  16.  “Number  of  error  messages  encountered  was:”  The  results  vary 
across  the  scale  and  show  no  tendency. 

Question  17.  “Recovering  from  errors  was:”  Results  vary  greatly  across  the  scale 
ranging  from  1  to  7.  No  tangible  patterns  can  be  seen. 

Learning.  The  questions  pertaining  to  the  ease  of  learning  how  to  use  the 
prototype  are  as  follows: 

Question  18.  “Learning  how  to  use  the  system  was:”  Results  show  and  overall 
positive  tendency.  Six  responses  are  positive. 

Question  19.  “Remembering  the  names  and  use  of  commands  was:”  All 
responses  are  positive.  One  response  is  rated  a  6  with  the  remaining  evenly  split  among  7 
and  8. 

Question  20.  “Exploration  of  system  functions  was:”  All  but  one  response  is 
positive.  The  responses  vary  relatively  evenly  across  the  positive  half  of  the  scale. 

Question  21.  “The  task  flow  between  the  real  worked  and  the  system  seemed  to 
be  the:”  Results  show  an  overall  positive  tendency  with  two  negative  responses.  The 
positive  responses  are  evenly  distributed  from  7  to  10. 
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Prototype  Capabilities. .  The  questions  pertaining  to  the  system  capabilities  of  the 
prototype  are  as  follows; 

Question  22.  “Response  time  from  the  system  was:”  All  responses  are  positive 
and  rated  at  7,  8,  or  9. 

Question  23.  “Overall,  use  of  the  system  was.”  All  responses  were  positive 
toward  the  system  being  reliable  with  five  responses  being  rated  8. 

ITI-ALC  open  end  questions.  Specific  open  end  questions  pertaining  to  the  ITI- 
ALC  prototype  are  as  follows: 

Question  24.  “What  did  you  particularly  like  about  ITI-ALC?”  This  question 
specifically  addresses  those  aspects  of  the  prototype  that  stood  out  and  were  appealing  to 
the  users.  Two  comments  identified  positive  reaction  to  the  system  providing  the 
information  needed  by  the  mechanic.  Four  responses  addressed  the  flow  of  information. 
The  first  being  that  a  system  such  as  ITI-ALC  would  make  it  easier  to  track  work  and 
parts.  The  second  indicated  the  flow  of  information  as  he  went  through  the  prototype  was 
“fairly  easy  with  minor  problems.”  The  third  and  fourth  responses  dealing  with  the  flow 
of  information  were  unclear  to  exactly  what  they  were  trying  to  say.  The  last  users 
comment  indicated  that  “his  grand  kids  will  love  it.”  The  exact  meaning  to  this  response 
could  take  several  avenues,  but  is  taken  as  being  a  positive  reaction  to  the  prototype. 

Question  25.  “What  did  you  particularly  NOT  like  about  ITI-ALC?”  This 
question  specifically  addresses  those  aspects  of  the  prototype  that  are  disliked  by  the 
users.  Three  comments  dealt  with  the  user  interface.  The  first  comment  suggested  that  it 
needed  to  be  more  user  friendly.  The  second  comment  said  that  some  commands  were  a 


little  misleading.  The  third  comment  indicated  that  some  T.O.  Steps  were  hard  to  follow 
with  not  enough  information  being  provided  to  make  instructions  clear.  Two  more 
responses  said  there  was  nothing  they  disliked  about  the  ITI-ALC  prototype  and  the  last 
suggested  that  he  would  be  dead  before  he  would  get  to  use  it. 

Question  26.  “What  recommendations  would  you  make  for  improving  ITI- 
ALC?”  This  question  seeks  insight  into  the  overall  evaluation  of  prototype.  Two  users 
suggested  that  more  studies  should  be  conducted  on  the  system  Avith  even  more  input 
coming  from  the  mechanic.  Two  other  responses  suggested  that  the  system  should  be 
made  more  user  friendly,  but  made  no  specific  suggestions  on  what  would  make  the 
system  more  user  friendly.  The  three  remaining  users  comments  dealt  with  receiving 
training  on  how  to  use  computers  and  the  system. 

Observations  noted  on  the  interaction  of  the  users  with  the  prototype  also 
contributed  information  to  investigative  question  1 .  The  observations  provided  details  to 
identify  specific  items  on  the  screens  that  created  confusion  or  problems.  Nothing  was 
identified  as  a  major  inhibitor  for  the  prototype.  The  observations  in  general,  support  the 
information  gathered  by  the  GSS  and  the  usability  questionnaire.  In  addition,  it  provides 
the  specific  location  of  many  of  the  problems. 

Invesigative  question  2 

If  the  prototype  does  not  meet  the  users  needs,  what  changes  or  modifications  need  to  be 
made  to  correct  the  prototype  system  so  that  it  does? 

Investigative  question  2  is  a  logical  follow  on  of  investigative  question  1 .  It 
identifies  the  deficiencies  that  make  the  prototype  not  completely  meet  the  users  needs. 
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As  in  investigative  question  1,  investigative  question  2  requires  the  use  of  all  three  of  the 
data  collection  methods:  the  GSS,  the  usability  questionnaire,  and  the  researcher 
observations. 

The  GSS  provided  the  majority  of  results  for  answering  investigative  question  2. 
These  results  were  summarized  in  Table  1  and  further  explained  in  the  section  that 
followed. 

The  usability  questionnaire  provides  information  that  identifies  general  areas  of 
the  prototype  that  could  be  improved  in  the  interface  development.  The  information  is 
not  as  specific,  but  does  describe  the  general  impression  of  the  users  and  identifies  those 
areas  where  specific  problems  were  not  addressed  in  the  other  two  data  collection 
methods. 

The  information  obtained  through  researcher  observation  also  identify  areas  for 
improvement.  Much  of  the  information  was  a  duplication  of  that  identified  using  the 
GSS.  The  ideas  that  did  not  duplicate  the  those  gathered  with  the  GSS  are  shown  in 
Table  5  and  should  be  included  with  the  GSS  information  to  create  the  complete  list  of 
suggested  changes. 

Table  5.  Observations  not  documented  in  GSS 


not : 

■v  .ooltecMInGSl  •••••• 

Meaning  of  status— wasn’t  sure  if  it  meant  completion  or  not _ 

Drop  Menu  for  login  name-immediately  provide  list  of  names _ 

Double  click  hides  schedule— A  general  problem  Min/Max  without  exec,  commands 
Step  check  boxes  capability  not  there— ’’Double  clicked  several  times”  Prototype  limit 

Wants  previous  write-ups  to  come  up  automatically  when  receiving  the  job _ 

Unclear  exactly  when  you  are  ready  to  start  a  task 

Screens  with  o  action--no  one  knows  where  to  proceed  unless  directed  to  on  screen 
Should  say  whether  or  not  a  job  is  supportable  or  give  reasons— also  give  delay  times 
Need  status  of  ordered  parts/equipment  or  task  can’t  be  done 


Table  5.  Observations  not  documented  in  GSS  (Continued) 


“173  card”  not  labeled— caused  participant  to  search  in  menu  for  the  173  card _ 

Difficulty  with  mechanic  list  box _ 

“Back”  label  on  locator  and  Full  Screen  may  be  confusing _ 

Master  schedule  wAVork  operation  completion  notice  should  be  the  model  for  screens 


Investigative  question  3. 

Does  the  ITI-ALC  system  present  a  “Value-added"  to  the  user? 

To  address  this  investigative  question,  the  result  obtained  on  questions  1,2,  and  24 
of  the  usability  questionnaire  are  revisited.  To  reiterate,  questions  1  and  2  both  received 
positive  responses.  Question  24  solicits  vt^hat  was  particularly  liked  about  the  prototype. 
The  results  are  as  follows:  Two  comments  identified  positive  reaction  to  the  system 
providing  the  information  needed  by  the  mechanic.  Four  responses  addressed  the  flow  of 
information.  The  first  was  that  a  system  such  as  ITI-ALC  would  make  it  easier  to  track 
work  and  parts.  The  second  indicated  the  flow  of  information  as  he  went  through  the 
prototype  was  “fairly  easy  with  minor  problems.  The  third  and  fourth  responses  dealing 
with  the  flow  of  information  were  unclear  to  exactly  what  they  were  trying  to  say.  The 
last  users  comment  indicated  that  “his  grand  kids  vwll  love  it.”  The  exact  meaning  to  this 
response  could  take  several  avenues,  but  is  taken  as  being  a  positive  reaction  to  the 
prototype. 

Investigative  question  4. 

Does  using  a  group  support  system  aid  in  prototype  analysis? 

This  investigative  question  seeks  to  find  out  if  using  the  GSS  was  efficient, 
effective,  and  would  the  users  use  it  to  perform  an  evaluation  such  as  this  again.  Three 
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questions  on  the  usability  questionnaire  address  this  issue.  The  responses  to  these 
questions  are  shown  in  Table  3.  Questions  27, 28,  and  29  used  a  5  point  agree/disagree 
scale  to  gather  user  responses.  A  value  of  1  was  assigned  to  the  strongly  disagree 
response  and  a  value  of  5  was  assigned  to  the  strongly  agree  response  with  the 
corresponding  integer  values  being  assigned  to  those  responses  falling  in  between.  The 
value  assignment  allowed  weighted  averages  to  be  calculated  which  could  be  used  to 
obtain  the  mean  and  the  standard  deviation  for  the  group. 

Question  27.  “I  would  use  a  Group  Support  System  to  evaluate  another 
prototype.”  All  responses  were  either  agree  or  strongly  agree. 

Question  28.  “The  Group  Support  System  helped  me  to  evaluate  the  rapid 
prototype  faster  than  if  I  didn’t  use  it.”  All  responses  were  either  agree  or  strongly  agree 
that  the  prototype  was  evaluated  faster  than  if  they  did  not  use  it. 

Question  29.  “The  Group  Support  System  was  effective  in  helping  me  evaluate 
the  rapid  prototype.”  Again,  all  responses  were  either  agree  or  strongly  agree. 

The  researcher  observations  contribute  information  that  can  be  used  to  answer  this 
investigative  question.  The  researcher  observations  provided  a  good  cross  checks  as  well 
as  an  alternate  source  of  information. 

Consolidated  notes  identified  63  observations  (Appendix  H).  Of  the  63 
observations,  9  were  also  collected  by  the  users  using  the  GSS.  Thirteen  more 
observations  were  not  documented  while  using  the  GSS  These  13  observation  are  listed 
in  Table  5.  Researcher  observations  also  identified  twenty  more  areas  where  lack  of 
training  on  the  prototype  created  problems.  These  twenty  are  not  necessarily  considered 
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functions  or  capability  of  the  prototype.  Of  these  20  training  areas,  16  were  not 
previously  identified  by  users  in  the  GSS. 

Summary 

Three  forms  of  data  were  collected  for  this  study:  group  prototype  evaluation 
results,  individual  prototype  usability  results,  and  observations  on  user-prototype 
interaction.  Seven  users  consisting  of  programmed  depot  maintenance  (PDM)  technicians 
and  functional  supervisors  were  asked  to  evaluate  the  ITI-ALC  prototype. 

GSS  results  show  that  18  ideas  were  collected.  Sixteen  of  these  were  suggested 
changes  or  modifications  and  2  were  positive  comments  about  the  prototype  opening 
screen  and  the  task  scenario  itself. 

The  usability  questiormaire  consisted  of  29  questions  that  addressed  different 
aspects  of  the  prototype  usability,  the  general  impression  of  ITI-ALC  as  portrayed  by  the 
rapid  prototype,  and  using  a  GSS  while  performing  the  evaluation.  Twenty  three  of  these 
questions  pertained  to  the  usability  of  the  prototype.  There  were  19  questions  with 
generally  positive  responses,  one  with  a  generally  negative  response,  and  three  questions 
that  the  responses  did  not  give  any  tendencies  either  positive  or  negative.  Collectively, 
responses  indicate  that  the  prototype  usability  was  an  83%  positive  and  4%  negative  as 
evaluated  by  the  users  in  this  study.  Three  open  end  questions  addressed  the  likes  and 
dislikes  of  ITI-ALC  as  a  potential  depot  information  system.  The  aspects  ITI-ALC  that 
the  users  liked  was  the  information  that  it  provided  to  the  mechanic  and  that  it  seemed  to 
enhance  the  information  flow  in  the  depot  process.  Dislikes  of  ITI-ALC  indicate  three  of 
the  users  did  not  like  certain  aspects  of  the  user  interface,  but  two  indicated  that  they 
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found  nothing  that  they  disliked  about  ITI-ALC.  Observations  of  the  interaction  of  the 
users  with  the  prototype  totaled  63.  Nine  were  duplications  of  information  collected 
using  the  GSS  and  thirteen  were  tmique  to  this  method  of  data  collection.  Twenty 
observations  could  be  attributed  to  lack  of  training  of  the  prototype.  The  remaining 
observations  fell  into  no  distinct  category  and  are  considered  as  general  comments. 

Investigative  questions  1  and  2  are  addressed  by  using  all  the  ideas  generated 
using  the  GSS,  the  first  26  question  asked  in  the  usability  questionnaire  and  selected 
noted  observations.  Investigative  question  3  is  addressed  by  using  the  responses  from 
three  of  the  questions  in  the  usability  questionnaire  and  selected  observations.  All  three 
yielded  generally  positive  responses.  Question  4  was  addressed  solely  by  the  last  three 
questions  of  the  usability  questionnaire.  The  three  questions  addressed  whether  the  GSS 
was  efficient,  effective,  and  would  the  users  use  the  GSS  again  to  perform  an  evaluation. 
The  responses  were  very  positive  to  each  question. 

Each  investigative  question  was  addressed  with  the  information  found  pertinent 
from  each  data  collection  method.  A  discussion  of  the  findings  will  follow  in  chapter  5 
of  this  thesis. 


64 


V.  Discussion.  Recommendations,  and  Conclusions 


Chapter  Overview 

This  chapter  contains  a  discussion  of  the  results  found  during  this  study.  The  discussion 
of  the  data  collected  addresses  the  four  investigative  questions  that  were  previously  identified. 
Limitations  of  this  study  are  discussed.  Recommendations  for  the  improvement  of  ITI-ALC 
prototype  as  well  as  prototype  evaluation  using  the  three  data  collection  methods  used  in  this 
study  are  also  discussed.  Recommendations  for  further  study  on  the  subject  and  associated 
topics  are  presented. 

Discussion  of  Results 

The  discussion  of  the  results  will  follow  each  stated  investigation  question.  Information 
obtained  using  each  instrument  is  combined  to  address  the  issues  of  each  investigative  question. 
Investigative  question  1 

The  first  investigative  question  asks,  “to  what  extent  has  the  current  prototype  adequately 
captured  the  user  requirements  for  managing  programmed  depot  maintenance  and  for  performing 
a  depot  maintenance  task?” 

Results  of  this  study  indicate  that  the  ITI-ALC  prototype  adequately  captures  the  users' 
requirements.  The  study  shows  that  users  found  no  major  requirement  oversights  in  the 
prototype  concerning  planning  or  performing  maintenance  tasks.  The  users  did  however,  submit 
several  recommendations  to  improve  the  prototype  during  the  evaluation. 
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Most  problems  identified  in  the  evaluation  appear  to  indicate  that  modification  of  the 
existing  screens  or  fields  within  those  screens  are  all  that  is  needed  to  make  them  satisfactory. 
Several  problems  noted  relate  to  the  ability  to  navigate  within  the  prototype.  The  majority  of  the 
navigation  problems  identified  appear  to  fall  into  3  main  categories.  The  first  area  is  the  way  the 
T.O.  Graphics  are  presented  in  the  electronic  T.O.’s.  The  second  is  the  users  lack  of  experience 
using  computers  or  a  Graphical  User  Interface  (GUI).  The  third  is  the  lack  of  prior  training  on 
using  the  ITI-ALC  prototype.  Another  problem  that  seemed  to  stand  out  was  the  users  confusion 
with  terminology  used  in  the  prototype 

Technical  Order  Graphics.  Depot  mechanics  use  diagrams  and  graphics  within  T.O.s 
during  maintenance  on  a  regular  bases.  The  evaluation  indicated  problems  with  the  T.O. 
Graphics  that  pointed  mostly  to  the  ability  for  the  user  to  see  the  whole  graphic.  For  example, 
the  window  showing  the  graphic  that  is  in  the  lower  half  of  the  screen,  appeared  to  be  too  small. 
In  addition,  the  scroll  bars  are  meant  to  allow  the  user  to  see  a  different  area  of  the  graphic 
seemed  frustrating  to  use.  Comments  were  made  such  as  a  menu  item  needs  to  be  developed  that 
allow  the  exploded  view  of  and  item  to  be  created.  Specific  items  pertaining  to  this  are  shown  in 
the  GSS  documentation,  the  usability  questionnaire,  and  the  throughout  the  observation  notes. 

Computer  experience.  Comments  made  by  users  during  the  evaluation  indicated  they 
would  like  more  training  on  computers.  An  example  of  the  problems  experienced  by  the  users 
was  icon  recognition.  Users  did  not  know  that  putting  the  cursor  on  the  icon  will  give  the  name 
of  its  function.  Observations  indicated  that  many  of  the  users  were  unfamiliar  with  the  standard 
windows  format  of  the  prototype  and  were  hesitant  to  explore  the  capabilities  and  functions.  The 
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users  were  unfamiliar  with  how  to  use  the  “pop  down”  menus,  edit  boxes  that  appeared  confused 
them,  double  clicking  the  mouse  was  new  to  them,  etc. 

ITI-ALC  Training.  Fimctions  of  the  ITI-ALC  prototype  were  not  intuitively  obvious  to 
the  users  by  looking  at  the  menu  items  of  the  prototype.  As  the  users  proceeded  through  the 
prototype  evaluation,  a  learning  curve  could  be  seen.  For  example,  most  of  the  users  had 
difficulty  logging  on  to  the  prototype  as  a  specific  user  at  the  beginning  of  the  evaluation.  By  the 
last  part  of  the  scenario,  this  problem  had  been  overcome  and  the  users  were  smoothly  logging 
off  and  on  the  prototype.  Users  also  stated  that  training  on  how  to  use  the  system  would  be 
necessary  for  the  success  of  an  ITI-ALC  system. 

Terminology.  Many  problems  noted  in  the  evaluation  of  the  prototype  revealed  that  a 
common  terminology  did  not  exist  between  the  what  the  designers  used  in  the  prototype  menu 
and  what  the  users  expected.  While  following  the  scenario,  difficulties  were  encountered  by  the 
users  because  they  were  searching  for  a  different  term  -within  the  menu.  As  a  result,  confusion 
was  created  for  the  user  as  to  which  item  to  select  to  perform  the  requested  action.  Some  users 
did  not  draw  the  parallel  meaning  between  “Logout”  and  “Sign  off’  for  example.  The  term  O- 
level  was  not  intuitive  to  some  of  the  users.  Many  instances  such  as  these  are  documented  both 
Table  1  (GSS  documented  information)  and  observation  notes  (Appendix  H).  Contradictorily, 
while  performing  the  evaluation,  terminology  seemed  to  be  a  problem,  but  the  terminology  was 
considered  positive  in  the  usability  analyses  questions. 
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Investigative  question  2 


Investigative  question  2  states  that  if  the  prototype  does  not  meet  the  users  needs,  what 
changes  or  modifications  need  to  be  made  to  correct  the  prototype  system  so  that  it  does?  As 
shown  above,  the  prototype  does  meet  user  needs,  so  this  question  does  not  need  to  be  addressed. 
However,  the  GSS  documented  information  in  Table  1  states  specifically  those  items  that  the 
users  felt  needed  to  be  changed  to  more  adequately  meet  the  depot  users  needs. 

The  usability  questionnaire  indicates  that  the  users  have  an  overall  positive  reaction  to  the 
prototype  in  the  areas  of  expectation,  screen  organization,  navigation,  terminology  use, 
minimizing  potential  errors,  learning,  and  the  prototype  capabilities.  A  specific  area  that  was 
identified  as  having  a  negative  reaction  by  the  group  was  the  prototype's  ability  to  provide 
instruction  on  how  to  navigate  from  screen  to  screen.  Details  that  support  this  result  are 
identified  in  the  GSS  Table  1  and  throughout  the  researcher  observation  notes.  Questions  25  and 
26  of  the  questionnaire  specifically  ask  for  what  the  users  did  not  like  about  the  prototype  and  the 
recommendations  they  had  for  improving  the  prototype,  respectively.  Users  said  that  designers 
should  try  to  make  the  prototype  more  user  friendly  and  menu  commands  more  clear.  They  also 
indicated  that  more  graphics  such  as  3  dimensional  or  exploded  views  should  be  provided  of  the 
areas  where  they  are  working.  Other  improvements  were  to  conduct  more  studies,  to  continue 
getting  feedback  from  the  mechanics  and  to  provide  training  for  the  users  on  ITI-ALC  and  the 
use  of  computers  in  general. 
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Table  5  lists  13  researcher  notes  on  areas  for  improving  the  prototype.  Substantial 
information  can  also  be  gleaned  by  reviewing  the  observational  notes  for  improving  the  ITI-ALC 
prototype. 

Investigative  question  3 

Investigative  question  3  asks,  “does  the  ITI-ALC  system  present  a  “Value-added”  to  the 
user?”  General  indications  of  the  results  of  the  study  indicate  that  ITI-ALC  is  a  “Value  added”  to 
the  user.  Questions  pertaining  to  the  overall  system  indicated  a  positive  reaction  by  the  users 
although  not  an  overwhelming  one.  Question  24  solicited  comments  about  what  the  users 
particularly  liked  about  the  prototype.  Specific  comments  from  the  user  on  this  question 
indicated  that  ITI-ALC  would  provide  the  information  needed  by  the  depot  mechanic.  They  also 
indicate  and  that  using  the  system  would  make  it  easier  to  track  parts  and  monitor  the  depot 
work  on  aircraft. 

Investigative  question  4 

Investigative  question  4  asks,  “does  using  a  Group  Support  System  aid  in  prototype 
analysis?” 

The  findings  in  this  study  indicate  that  using  a  GSS  does  support  prototype  analysis. 
Results  show  that  all  users  believe  that  the  GSS  was  effective  and  aided  in  helping  to  evaluate 
the  prototype  faster  than  if  they  did  not  use  it.  They  also  unanimously  agreed  that  they  would 
use  a  GSS  to  evaluate  a  prototype  again. 

Additionally,  this  study  identified  areas  where  the  GSS  did  not  support  the  prototype 
analysis  well.  These  areas  are  when  the  users  were  unskilled  in  typing,  and  when  problems  were 
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difficult  for  the  users  to  describe  in  words  or  they  did  not  realize  that  a  problem  existed.  Specific 
items  that  were  not  documented  using  the  GSS  but  were  identified  by  one-on-one  researcher  and 
user  interaction  are  in  Table  5. 

Typing  skill.  This  researcher’s  observations  were  that  skilled  typing  users,  appeared  to  use  the 
GSS  more  to  document  the  findings  of  the  evaluation.  When  the  users  were  unskilled  in  typing, 
they  seemed  to  want  to  verbalize  their  problems  to  the  researcher  as  opposed  to  typing  the 
problem  into  the  GSS.  In  many  cases  they  seemed  very  reluctant  to  type  the  information  into  the 
GSS,  even  though  they  felt  the  information  was  valuable.  To  alleviate  this  problem,  during  the 
brainstorming  portion  of  the  evaluation,  a  more  skilled  typist  would  enter  the  information  into 
the  GSS.  After  such,  the  user  that  initiated  the  idea  would  confirm  that  the  information  was 
correct. 

Documentation  difficulties.  In  some  cases,  the  users  had  difficulty  putting  into  words  the 
difficulty  they  were  experiencing.  In  cases  such  as  this,  information  is  most  certainly  lost  when 
solely  using  the  GSS  because  the  user  might  elect  to  not  document  the  problem. 

T  Jnaware  of  problem.  In  some  cases  the  users  did  not  realize  they  were  having  a  problem.  They 
seemed  to  believe  that  the  problem  was  in  their  own  ability  and  not  in  the  instructions  of  the 
prototype.  In  situations  like  this,  the  GSS  would  not  adequately  capture  the  problems  and  the  use 
of  an  observer  might  be  the  best  way  to  capture  the  information. 

Limitations  of  this  study 

Simulated  environment 
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This  is  only  one  study  conducted  using  a  prototype.  As  a  simulation  with  a  prototype,  it 
may  not  totolly  match  the  experience  of  real  work  with  an  operational  system.  The  laboratory 
environment  controls  many  of  the  elements  that  can  effect  an  operation  system.  The  closer  to  the 
actual  environment  the  more  realistic  the  test  is  and  therefore  the  better  the  results  are. 

Computer  and  typing  skills.  Two  of  the  largest  limitations  of  this  study  were  the  effects 
that  limited  computer  experience  and  lack  of  typing  skills  had  on  using  the  GSS.  The  pairing  of 
these  two  limitations  may  have  limited  the  number  of  ideas  generated  during  evaluation  of  the 
prototype  for  some  users.  It  also  served  to  extend  the  time  required  by  some  to  perform  the 
evaluation  and  complete  the  associated  questionnaires.  The  fhistration  this  created  for  the  users 
could  possibly  have  inhibited  the  perceived  effectiveness  of  the  GSS  in  the  prototype  evaluation. 
When  dealing  with  users  that  are  very  comfortable  using  computers  and  associated  software,  the 
GSS  can  be  a  tremendous  asset.  When  this  capability  does  not  exist,  or  is  limited,  the  one-on- 
one  approach  of  prototype  evaluation  is,  in  this  researcher’s  opinion,  the  preferred  evaluation 
method. 

GSS  Laboratory.  Another  limitation  of  this  study  was  the  size  and  environmental 
conditions  of  the  room  where  the  prototype  evaluation  took  place.  The  room  dimensions  were 
small  and  limited  the  “elbow  room”  that  each  user  had  to  work.  Compounding  this  limitation 
was  that  the  air  conditioning  in  the  building  was  not  working  properly,  making  the  room 
uncomfortable  to  work  in  for  an  extended  period  of  time.  The  combining  of  these  two 
limitations  along  with  the  extra  time  required  to  conduct  the  evaluation  because  of  limited 
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computer  and  typing  skill,  imdoubtedly  had  a  negative  effect  on  the  volume  of  ideas  as  compared 
to  if  these  limitations  did  not  exist. 


Recommendations  for  Further  Research 

Two  areas  for  further  research  have  been  identified  from  this  study.  The  first  is  to  make 
the  suggested  changes  in  the  ITI-ALC  prototype  and  conduct  further  studies  and  evaluations. 

The  second  is  to  initiate  further  research  into  using  a  GSS  to  perform  prototype  analysis. 

More  research  on  the  ITI-ALC  prototype  should  be  conducted  before  the  final 
demonstration  system  configuration  is  decided.  “It  is  impossible  to  develop  a  solution  to  solve  a 
design  problem  in  a  single  iteration”  (Gould  &  Lewis,  1985:302).  Further  study  should  identify 
other  avenues  to  further  improve  and  refine  the  ITI-ALC  system.  The  iterative  process  of 
developing  and  testing  the  prototype  has  proven  to  be  winning  approach  for  Armstrong 
Laboratory  and  continuation  of  this  approach  is  warranted.  The  suggested  changes  put  forth  by 
the  users  should  be  reviewed  as  well  as  the  data  collected  during  the  study  that  does  not  identify 
specific  changes.  This  may  facilitate  more  changes  that  will  be  beneficial  to  the  ITI-ALC 
demonstration  system.  After  changes  have  been  made  to  the  prototype,  one  to  two  more 
evaluations  should  be  conducted  before  establishing  the  demonstration  system  final  design. 

Modifications  to  the  current  user  centered  workshop  plan  should  include  a  better 
assessment  of  the  user’s  computer  skills  and  the  modification  or  elimination  of  pre-determined 
human  factors  categories  used  in  the  evaluation. 
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Pre-determined  categories  during  prototype  evaluation  can  be  beneficial  when  the  terms 
and  categories  are  understood  by  all  that  are  using  them.  When  they  are  not,  it  can  be  a  source  of 
confusion  and  the  results  may  be  disappointing.  Whether  or  not  to  use  pre-determined  human 
factors  categories  should  be  dictated  by  users  performing  the  evaluation. 

Research  should  be  conducted  to  determine  more  specifically  the  benefits  of  using  a  GSS 
to  facilitate  the  evaluation  of  prototypes.  Guidelines  for  when  to  use  a  GSS  for  prototype 
analysis  should  also  be  developed.  A  GSS  according  to  the  findings  of  this  study  can  be  a  help 
in  prototype  evaluations.  A  study  should  also  be  conducted  to  perform  a  direct  comparison 
between  using  a  GSS  to  perform  a  prototype  analysis  and  using  the  one-on-one  approach 
typically  adopted. 

Conclusions 

The  ITI-ALC  rapid  prototype  appears  to  functionally  meet  the  requirements  of  the  users 
and  it  does  present  “value  added"  information.  Several  suggestions  however,  were  proposed  by 
the  users  performing  the  evaluation.  The  suggested  modification  or  changes  generated  by  the 
users  will  help  improve  the  prototype,  and  as  a  result,  gain  more  user  acceptance.  According  to 
Nielson  and  Levy,  user  feedback  and  opinions  are  valuable  data  that  should  be  used  in  the 
development  or  choosing  of  a  user  interface  design.  The  user  interface  of  a  system  or  software 
package  has  a  very  good  chance  of  being  successful  if  it  agrees  with  the  interface  opinions  of  the 
users  (Nielson  and  Levy,  1994). 

Results  of  this  study  also  indicate  that  a  GSS  is  useful  in  performing  prototype  analysis. 
Users  found  that  the  GSS  was  both  effective  and  efficient  to  use  while  performing  the  prototype 
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analysis.  They  also  indicated  that  they  would  use  the  GSS  again  to  perform  another  prototype 
analysis  given  the  opportunity. 
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Appendix  A:  User  Centered  Workshop  Plan 
I.  User  Centered  Workshop  Plan 


Purpose 

The  objectives  of  this  thesis  research  are  to:  (1)  perform  an  assessment  of  the  ITI-ALC 
rapid  prototype  and  elicit  suggestions  for  changes  or  modifications,  to  the  current  system 
requirements  that  are  necessary  for  the  initial  baseline  of  the  ITI-ALC  system,  (2)  determine 
the  ITI-ALC  rapid  prototype’s  compatibility  with  the  user’s  needs,  and  (3)  investigate  the  use 
of  group  support  systems  for  prototype  analysis. 

Hardware 


The  ITI-ALC  rapid  prototype  will  be  presented  on  386SX  25  Mhz  computers  in  the 
groupware  laboratory  located  in  room  319  of  building  641,  at  the  Air  Force  Institute  of 
Technology  (AFIT). 

Software 


The  software  for  this  study  includes  the  rapid  prototype  developed  by  Armstrong 
Laboratories  using  Microsoft  Visual  Basic  3.0  Professional  and  Microsoft  Access  2.0,  and 
Ventana  Group  System  for  Windows  that  will  facilitate  evaluation  data  collection. 

Room  Equipment 

The  room  will  be  equipped  with  8  users  user  terminals  and  1  facilitator  terminal  networked 
together.  It  will  also  have  an  overhead  projector  equipped  with  LCD  panel  for  displaying 
information  to  the  group  and  two  white  boards  to  aid  potential  group  discussions. 


Subjects 

There  will  be  a  total  of  7  aircraft  programmed  depot  maintenance  (PDM)  technicians, 
functional  supervisors  or  managers  for  the  aircraft  depot  line  at  Warner  Robins  AFB  Air 
Logistics  Center.  All  users  will  be  either  currently  performing  the  job  they  represent,  or  have 
recent  experience  in  that  position. 

Task 


Each  user  will  review  the  ITI-ALC  rapid  prototype  by  stepping  through  an  established 
scenario  (Appendix  B).  Users  will  be  firee  to  converse  with  each  other,  if  they  wish,  as  they 
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perform  the  walkthrough  thereby  taking  advantage  of  the  individual  evaluation  and  group 
interaction.  As  user  perform  the  walkthrough,  they  will  document  their  comments  using  a  group 
system  software.  The  group  system  software  used  for  this  study  is  the  Ventana  GroupSystem  for 
Windows.  This  software  will  serve  as  a  tool  to  collect  and  present  ideas  for  improving  or 
modifying  the  rapid  prototype.  These  ideas  will  subsequently  be  rated  on  a  scale  fi-om  1  to  10  by 
each  individual.  The  rating  results  will  then  be  combined,  creating  an  initial  ranking  of  all  the 
group  ideas.  This  experiment  will  provide  valuable  feedback  on  the  rapid  prototype  by  providing 
suggested  areas  for  improving  the  system’s  usability  and  enhancement  of  the  requirements 
already  addressed  by  the  prototype. 

Conditions 


The  rapid  prototype  will  be  presented  independently  on  each  users  terminal.  The  users,  made 
up  of  mid-level  managers  and  maintenance  technicians  from  one  base,  will  simultaneously 
provide  comments  on  the  prototype  using  group  software.  This  simultaneous  evaluating  of  the 
rapid  prototype  and  documenting  the  ideas,  will  be  possible  by  having  both  programs  running 
and  using  the  “Alt-tab”  function  to  toggle  between  the  two.  The  group  software  allows  users  to 
document  and  interactively,  yet  anonymously,  discuss  their  ideas  for  improving  the  rapid 
prototype.  The  group  software  allows  each  user  to  view  their  own  ideas,  as  well  as  those  made 
by  the  other  users  on  their  own  computer  terminal  or  on  a  separate  display  overhead,  aptly  named 
the  “Big  Screen”. 

Projected  Study  Results 

1.  Improved  xmderstanding  of  ITI-ALC  suitability  in  satisfying  user  needs  by  identifying 
those  rapid  prototype  features  that  worked  well,  and  those  did  not. 

2.  Feedback  from  the  users  in  this  study  will  provide  insight  into  the  changes  that  should  be 
made  to  the  evolving  design  concepts  for  the  ITI-ALC  system  development  and  those  features 
which  currently  meet  or  exceed  user’s  needs. 

3.  The  ideas  collected  in  this  study  will  made  available  to  Armstrong  laboratory  to  provide 
to  the  contractor  developing  the  ITI-ALC  demonstration  system. 

4.  This  study  will  illustrate  the  potential  value  of  groupware  in  future  usability  studies. 

Data  Collected 


Demographic  data  will  be  collected  for  each  individual  using  the  Personal  Background  Form 
(Appendix  C).  During  the  evaluation,  ideas,  comments,  categories,  and  ratings  made  by  the 
users  will  be  collected  using  the  group  software.  Evaluator  observations  in  the  form  of  notes  and 
comments  on  a  paper  copy  of  the  walkthrough  will  also  be  collected  during  the  session. 
(Appendix  H).  Data  produced  by  this  study  include  a  list  of  requirements,  changes  and 
modifications  to  be  considered  for  the  ITI-ALC  system.  For  each  change  or  modification 
identified,  the  users  will  provide  comments  and  prioritization  rating  (i.e.,  how  important  is  the 
requirement)  (Nielson,  1992). 
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A  questionnaire  (Appendix  D)  will  be  administered  at  the  conclusion  of  the  test  to  gather 
any  remaining  comments  or  suggestions  pertaining  to  the  rapid. 

Controls 


The  goal  of  this  study  is  to  gather  iminhibited  ideas  on  a  rapid  prototype  to  identify  missing 
requirements  and  usability  problems  of  the  ITI-ALC  system  prior  to  its  final  design.  The  nature 
of  this  study  is  to  therefore  have  limited  controls  and  to  place  minimal  restrictions  on  the  group. 
Some  controls  however  are  unavoidable  in  order  to  maintain  the  integrity  of  the  results.  These 
controls  are  as  follows: 

1.  All  users  will  be  from  one  base  to  eliminate  variability  in  accepted  local  maintenance 
operating  procedures. 

2.  The  group  will  walk  through  a  single  scenario  during  the  initial  evaluation. 

3.  The  Grroup  software  laboratory  will  be  isolated  from  outside  activities  to  keep 
distractions  to  a  minimum. 


II.  Workshop  Procedures 


Introduction 


The  day’s  session  will  begin  by  introducing  the  users  to  the  purpose  of  the  study,  what  the 
ITI-ALC  rapid  prototype  is,  and  explaining  what  their  role  is  in  the  evaluation.  The  facilitator, 
technographer,  and  other  researchers  will  then  be  introduced  with  a  short  description  of  their 
roles.  Each  user  will  also  introduce  themselves  and  give  a  description  of  the  job  they  perform. 
The  logical  outcome  of  the  introduction  is  not  merely  to  introduce  the  people  to  each  other,  but 
rather  to  help  focus  on  what  the  problem  at  hand  is  and  begin  building  a  comfortable  working 
environment.  At  the  end  of  the  introduction,  users  will  human  use  and  consent  form  (Appendix 
I)  indicating  that  they  are  willing  to  participate  in  the  evaluation. 

Phase  I 

In  Phase  I,  the  users  are  introduced  to  the  Group  System  software,  which  will  be  the  primary 
tool  for  gather  information.  The  user  are  instructed  on  what  the  “buckets”,  “ideas”  and 
“comments”  functions  of  the  software  are  used  for.  Buckets  are  the  categories  that  the  ideas  and 
comments  will  be  put  into  during  the  evaluation.  They  are  typically  simply  a  general  heading. 
Buckets  for  screen,  navigation,  terminology,  and  errors  will  be  created  at  this  time  to  show  the 
users  how  to  create  buckets  so  that  more  can  be  added  during  the  evaluation  as  they  are  needed. 
Ideas  in  this  study,  will  be  the  major  point  or  problem  that  is  being  identified  about  some  aspect 
of  the  prototype.  Comments  can  be  considered  sub-bullets  or  text  to  help  clarify  the  meaning  or 
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provide  recommendations  for  the  basic  idea.  These  functions  will  facilitate  the  initial 
information  gathering  and  brainstorming  that  will  generate  ideas  during  this  evaluation. 

Phase  II 


After  the  group  software  tutorial  is  performed  and  the  users  are  comfortable  with  the  Group 
System  software,  the  group  will  proceed  to  the  beginning  of  the  evaluation  and  the  walkthrough 
scenario  of  the  ITI-ALC  rapid  prototype.  To  ensure  a  clear,  conceptual  understanding  of  how  the 
prototype  scenario  will  flow,  a  logic  flow  chart  depicting  a  typical  job  assignment  through 
completion  will  be  shown.  The  group  vsdll  now  begin  the  walkthrough,  ask  questions,  and  in 
general  get  a  good  “feel”  for  the  prototype  and  its  capabilities  as  well  as  its  functional  intent.  As 
the  users  walkthrough  the  scenario,  the  will  make  comments  on  the  rapid  prototype  using  the 
group  software.  Again  this  action  can  be  performed  by  using  the  “Alt-tab”  feature.  The  users 
will  create  a  list  of  ideas  identifying  changes,  modification  and  missing  requirements  pertaining 
to  the  rapid  prototype.  They  will  also  identify  those  features  of  the  rapid  prototype  that  they  like. 
These  ideas  will  be  presented  on  the  “big  screen”  as  well  as  the  terminal  of  each  user.  The 
facilitator  and  evaluators  from  Armstrong  laboratory  Avill  observe  any  problems  or  reactions 
toward  the  rapid  prototype  and  document  them  accordingly.  They  will  also  answer  questions  and 
provide  assistance  to  the  users  when  needed. 

Phase  III 


Once  the  all  users  have  completed  the  walkthrough,  and  comments  have  ceased,  the  group 
'will  review  as  a  group  the  ideas  that  have  been  generated.  The  group  will  discuss  the  ideas  and 
comments  making  additions  or  modifications  as  needed.  They  will  also  place  all  the  ideas  into 
groups  or  “Buckets”  which  identifies  the  common  category  that  each  of  the  ideas  are  in.  Initially 
common  groups  for  Screen,  Prototype  Navigation,  Terminology,  Errors,  and  Learning  problems, 
will  be  established  but  the  group  may  add  any  other  bucket  that  they  wish  to  group  the  ideas  as 
they  see  fit.  At  any  time,  users  can  refer  to  the  rapid  prototype  on  their  individual  terminal. 

They  can  also  request  to  see  a  rapid  prototype  screen,  series  of  screen,  or  the  walkthrough  in 
whole  or  in  part  on  the  big  screen  The  assimilation  of  free  flowing  ideas  and  comments  about  the 
protot5q)e  will  continue.  When  the  input  ideas  or  participation  reduces  to  a  level  where  the 
session  is  becoming  stagnant,  the  experimenter  will  attempt  to  reinitiate  the  group  interaction.  A 
experimenter  will  attempt  to  stimulate  discussion  in  an  area  that  has  not  received  much,  if  any 
attention.  This  process  will  continue  imtil  all  ideas  and  comments  are  exhausted. 

Phase  IV 


Phase  four  of  the  evaluation  will  be  the  rating  of  the  ideas  by  each  of  the  users.  To  prepare 
for  the  rating,  all  ideas  will  be  flattened.  In  flattening,  all  ideas  are  taken  out  of  the  buckets  and 
set  so  that  no  one  idea  is  grouped  or  deemed  more  important  than  the  other.  Now  the  rating  can 
begin. 
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In  rating,  users  will  have  a  rating  scale  to  the  right  of  each  idea.  Each  user  will  rate  the  idea 
according  to  the  severity  of  the  problem,  10  being  the  most  severe  and  1  being  the  least  severe. 
When  all  users  are  finished  rating  the  ideas,  the  Group  System  software  will  combine  each  users 
results  into  an  aggregate  ranking.  This  aggregate  ranking  will  represent  the  groups  ranking  of 
ideas  according  to  their  severity.  If  there  is  a  general  consensus,  the  rating  will  be  complete.  If  it 
is  not,  and  some  idea  ratings  appear  to  be  bi-polar,  the  group  will  discuss  the  ideas  with  the  most 
inconsistency  to  determine  why  this  might  be  the  case.  If  for  some  reason  a  user  wants  to  change 
his/her  rating,  they  may  modify  the  individual  ratings  accordingly. 


User  Feedback/Wrap-up 

At  the  conclusion  of  the  test,  a  questionnaire(Appendix  D)  will  be  administered  to  gather 
any  remaining  comments  or  suggestions  pertaining  to  the  rapid. 

Evaluation  measurement 

The  following  information  will  be  used  by  the  experimenter  to  analyze  and  draw 
conclusions  in  this  experiment. 

1 .  A  ranked  list  of  ideas  identifying  potential  changes  to  the  rapid  prototype. 

2.  The  consolidated  ideas  will  be  categorized  into  groups  or  areas  of  concern. 

3.  Explanatory  comments  on  each  idea  will  help  clarify  the  meaning  and  provide 
recommendations  on  missing  human-computer  interface  or  functional  requirements. 

4.  Individual  assessment  of  the  prototype  as  obtained  in  the  usability  questionnaire. 
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III.  Conducting  the  Workshop 


Sequence  of  Events 

•  Introduction  to  the  experiment 

•  Background  briefing  on  ITI-ALC 

•  Completion  of  the  Personal  Background  Form 

•  Introduction  to  the  group  software 

•  Rapid  prototype  walkthrough  and  initial  gathering  of  ideas 

•  Group  review  and  brainstorming  of  ideas  and  comments 

•  Consolidating  and  refining  of  the  group  comments 

•  Voting  and  rating  of  the  group  identified  deficiencies  by  the  individual  users 

•  Consolidated  rating  of  the  ideas 

•  Group  consensus  review  of  results 

•  Feedback/Wrap-up 

Introduction 


The  experiment  will  begin  with  an  introduction  of  the  researchers  and  their  roles. 
Specifically,  to  identify  the  study  facilitator,  technographer  and  other  researchers  that  are  present. 
The  users  will  then  introduce  themselves  and  identify  their  jobs  or  positions.  This  will  serve  to 
identify  those  participating  in  the  experiment  and  begin  to  build  group  familiarity.  The  users  will 
next  receive  a  briefings  that  outline  the  ITI-ALC  program,  describing  the  purpose  of  the 
experiment,  the  way  the  information  will  be  collected,  and  preliminary  instructions.  Preliminary 
instructions  will  include  the  responsibility  of  the  test  user,  the  type  of  information  that  will  be 
collected  during  the  experiment,  and  how  the  collected  data  will  be  used.  The  users  will  be 
reminded  that  their  participation  is  voluntary  and  the  data  collected  will  not  be  associated  with 
their  name. 


Group  software  tutorial 

The  users  will  be  introduced  to  the  group  software  and  receive  a  tutorial  on  how  to  use  it. 
This  tutorial  will  encompass  the  concepts  of  what  buckets,  ideas,  and  comments  are  and  how  to 
create  and  use  them  to  collect  the  information  on  the  rapid  prototype  during  this  study. 


Rapid  Prototype  Evaluation 

The  evaluation  will  begin  with  the  users  using  a  scenario  to  walkthrough  the  rapid  prototype 
and  use  group  software  to  collect  all  the  ideas  and  comments  that  are  generated. 
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The  users  will  then  transition  into  a  group  review  and  hrainstorming  session  on  the  rapid 
prototype  deficiencies.  In  this  session,  the  group  will  add  to,  modify,  regroup,  or  delete  buckets, 
ideas,  or  comments  as  needed  to  fully  dociunent  any  concerns  about  the  prototype.  The  group 
may  refer  to  the  rapid  prototype,  use  the  white  boards  or  use  any  other  tool  available  to  achieve 
their  end  results. 

Now  that  the  comments  are  refined,  they  will  be  removed  from  their  respective  categories, 
and  each  users  will  vote  on  the  severity  or  importance  of  each  noted  discrepancy.  The  group 
software  will  be  used  to  consolidate  and  present  the  results  of  the  individual  ratings.  Any 
discussions  and  changes  to  the  ratings  will  be  made  as  a  group  at  this  point.  The  result  ’will  be  a 
general  group  consensus  as  to  the  ranking  of  the  discrepancies. 

User  Feedback/ Wrap-up 

At  the  conclusion  of  the  test,  a  questionnaire  will  be  administered  to  gather  any  remaining 
comments  or  suggestions  pertaining  to  the  rapid. 
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Appendix  B;  Rapid  Prototype  Evaluation  Scenario 


Taskl 

You  are  the  manager  of  two  aircraft.  You  are  in  charge  of  PDM  requirements  for  each  of  these 
aircraft  from  Receiving  to  Delivery.  You  need  to  assign  Tim  T.  Toolman  to  Replace  the  FCR 
Left  Equipment  Rack  Support  Assembly  on  an  F-16. 

Start 

•  First,  log  in  as  the  Aircraft  Manager 

•  Your  first  screen  is  the  depot  floor  showing  the  aircraft  that  you  are  in  charge  of. 

Select  Aircraft  for  Maintenance  and  View  Aircraft  Schedule 

•  Go  to  the  schedule  for  the  F-1 6  that  you  are  in  charge  of 
Assign  Job  to  Mechanic 

•  Change  the  tasking  (Replacing  the  FCR  Left  Equipment  Rack  Support  Assembly)  from  Dan 
Carlson  to  Tim  T.  Toolman. 

Transmit  Assignment  Message 

•  Transmit  tasking  changes  to  personnel. 

•  Sign  off  as  the  Aircraft  Manager 
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Task  2 


You  are  a  mechanic  named  Tim  T.  Toolman  and  are  signing  onto  the  ITI-ALC  system  for  the 
day.  You  will  need  to  review  any  previous  write-ups  relating  to  the  FCR  rack,  get  ready  to  do 
maintenance,  and  replace  the  FCR  Left  Equipment  Rack  Support  Assembly. 

Start 

•  First,  log  in  as  the  mechanic 
Review  Job  Assignment 

•  Your  first  screen  is  the  Work  Operations  List 
Review  Previous  Write-ups 

•  Review  previous  write-ups  relating  to  the  FCR  Left  Equipment  Rack  Support  Assembly 

Get  Ready  to  do  Maintenance 

•  Get  the  technical  data  for  the  FCR  task  (over  the  RF  Modem  to  your  computer) 

•  Check  the  availability  of  AGE  for  your  task 

•  Check  to  see  if  you  have  all  the  parts  you  need  to  perform  the  task 

•  If  not,  then  request  the  needed  parts 

Perform  Maintenance 

•  Step  through  the  input  conditions  for  Replacing  the  FCR  Left  Equipment  Rack  Support 
Assembly. 

•  Indicate  that  the  removal  of  the  Dual  Mode  Transmitter  has  been  completed. 

•  When  you  get  to  the  first  step  of  the  task,  assume  you  do  not  know  where  this  FCR  rack  is 
and  look  at  a  graphic  identifying  the  access  door  for  this  FCR  rack. 

•  Indicate  that  you’ve  completed  the  Electrical  Surface  Bonding  Procedures. 
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•  Move  to  the  next  series  of  steps. 

•  At  step  3.  In  order  to  display  an  enlarged  view  of  the  rack,  find  the  hot  spot  on  the 
graphic. 

•  At  step  6.  On  a  graphic,  find  the  indication  of  the  gap. 

•  Move  through  the  next  series  of  steps 

•  Redisplay  the  Caution  and  Note  for  Step  7 
Complete  Job  Documentation 

•  Move  to  the  Completion  Form  (Similar  to  173  card) 

•  Assume  another  mechanic  needed  help  moving  a  cart  sometime  during  the  maintenance 
process.  You  lost  15  minutes  helping  out.  Make  a  note  of  this  delay. 

•  One  person  will  be  required  to  inspect  the  replacement 

•  Click  in  the  signature  box  (THIS  IS  A  SIMULATION  OF  SIGNATURES) 

Transmit  Repair  Information 

•  Notify  your  manager  of  job  completion 

•  Assume  it  is  now  lunch  time.  Sign  off  as  the  Mechanic 
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Task  3 


You  are  the  Aircraft  Manager  again.  You  want  to  check  to  see  if  Tim  has  completed  his  work. 

Start 

•  First,  log  in  as  the  Aircraft  Manager 

Review  Repair  Information 

•  Go  to  the  schedule  for  the  F-1 6  you  are  in  charge  of 

•  Check  the  form  Tim  filled  out 

•  Look  at  his  Start  and  Stop  times 

Revise  Schedule 

•  Revise  your  maintenance  schedule 

•  Sign  off  as  the  Aircraft  Manager 
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Appendix  C;  Personal  Background  Form 


1.  User  Name _ 

2.  Check  one: 

3.  Military _ 

4.  Civilian _ 

5.  Time  in  Service: _ (years/months) 

6.  Paygrade: _ 

7.  Current  occupational  specialty  (job  title  or  AFSC); 


Prior  Work  Experience: 

8.  Occupational  Specialties; 

9.  Where: 

10.  Number  of  years: 

11.  Dates 'worked: 


(1)  (2)  (3) 


Education/T  raining 

12.  Education  -  highest  grade  completed: _ 

Training  Courses  for  occupation 

13.  Courses:  _  _  _ 

14.  Number  of  years:  _  _  _ 

15.  Dates  worked:  _  _  _ 

16.  Aircraft  Depot  Experience: _ 

Computer  Experience 

17.  Indicate  models  or  types  of  systems  used  (Mainframe,  PC,  etc.)  and  years/months  of 

experience  with  those  systems _ 

Software  use 


Example:  (word  processor,  spreadsheet, )(Windows,  DOS,  Macintosh) 
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USER  EXPECTATIONS 

Frastrating  Satisfying 

1.  Overall,  use  of  the  system  was  0000000000 

Difficult  Easy 

2.  Overall,  use  of  the  system  was  0000000000 

Never  Always 

3.  The  needs  of  both  experienced  and  0000000000 

inexperienced  users  was  considered 

SCREEN 

Hard  to  read  Easy  to  read 

4  The  characters  on  the  computer  0000000000 

screen  were 

Disorganized  Organized 

5  The  sequence  of  information  on  the  0000000000 

screen  was 

NAVTGATTON 

Difficult  Easy 

6.  Logging  into  the  system  and  0000000000 

starting  the  task  was 

Illogical  Logical 

7.  Navigation  among  items  (e.g.  fill-ins)  0000000000 

on  the  screen  was 

Not  helpful  Helpful 

8.  Instructions  on  how  to  navigate  from  0000000000 

screen  to  screen  was 

Illogical  Logical 

9.  The  sequence  of  flow  from  one  screen  OOOOOOOOOO 

to  the  next  was 
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1 0.  Logging  off  the  system  was 
TERMTNOT.OGY 

1 1 .  The  terminology  used  throughout 
the  system  was 

12.  In  relation  to  the  work  you  do, 
the  use  of  terminology  was 

13.  The  terminology  used  in  messages 
was 

ERRORS 


14.  Correcting  mistakes  was 

15.  The  number  of  ITI-ALC  system 
failures  encountered  was 


16.  Number  of  error  messages 
encountered  was 


17.  Recovering  from  errors  was 

LEARNING 

1 8.  Learning  how  to  use  the 
system  was 

19.  Remembering  the  names  and  use 
of  commands  was 

20.  Exploration  of  system  functions 
was 


Difficult 

Easy 

0  0 

0 

0 

0 

0 

0 

0  0  0 

Hard  to  understand 

Easy  to  understand 

0  0 

0 

0 

0 

0 

0 

0  0  0 

Inconsistent 

Consistent 

0  0 

0 

0 

0 

0 

0 

0  0  0 

Not  helpful 

Helpful 

0  0 

0 

0 

0 

0 

0 

0  0  0 

Difficult 

Easy 

0  0 

0 

0 

0 

o 

0 

0 

0  0 

1  2 

3 

4 

5 

6 

7 

8 

9  More 

0  0 

0 

0 

0 

0 

0 

0 

0  0 

1  2 

3 

4 

5 

6 

7 

8 

9  More 

0  0 

0 

0 

O 

0 

0 

0 

0  0 

Easy 

Difficult 

0  0 

0 

0 

0 

0 

0 

0 

0  0 

Easy 

Difficult 

0  0 

0 

0 

o 

0 

0 

0 

0  0 

Difficult 

Easy 

0  0 

0 

0 

o 

0 

0 

0 

0  0 

Rigid 

Flexible 

0  0 

0 

0 

0 

0 

0 

0 

0  0 
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Different  Same 

21.  The  task  flow  between  the  real  world  0000000000 
and  the  system  seemed  to  be  the 

SYSTEM  CAP  ABIT  JTIRS 

22.  Response  time  from  the  system 
was 

23.  Overall,  the  system  was 

ITT-AT.C 

24.  What  did  you  particularly  like  about  ITI-ALC? 

25.  What  in  particular  did  you  NOT  like  about  ITI-ALC? 

26.  What  recommendations  would  you  make  for  improving  ITI-ALC? 

CiSS 

SA  A  N  D  SD 

27.  I  would  use  a  Group  Support  System  0  0  0  0  0 

to  evaluate  another  prototype. 

SA  A  N  D  SD 

28.  The  GSS  helped  me  to  evaluate  the  0  0  0  0  0 

rapid  prototype  faster  than  if  I  didn’t 

use  it. 

SA  A  N  D  SD 

29.  The  GSS  was  effective  in  hlping  me  0  0  0  0  0 

evaluate  the  rapid  prototype. 


Slow  Fast 

0000000000 

Unreliable  Reliable 

0000000000 
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Appendix  E;  Idea  Voting  Results 


Nuittber  of  in  I 


10  9  8 


1 .  T.O.  graphics 
insufficient 


2.  T.O.  presentation 


3.  Review  previous 
writeup 


4.  Aircraft  operation 
status 


5.  Aircraft  identification 


6.  Bar  graph  colors  need 
to  be  explained,  ex.  red- 
behind  schedule, 
(legend) 


7.  Get  ready  to  do 
maintenance 


8.  Planning  interface 
requirement 


9.  Should  be  able  to 
return  to  T.O..  task  when 
documenting  maint. 


10.  Group  write  ups  with  0 
work  areas 


11. 173  cards  need  to  be 
shown  on  screen  as  a 
number 


12.  Support  equipment 


13.  Perform  maint 


14.  Age  equipment 
backorder  unclear 


15.  Closing  out  not  clear  0 
from  task  to  task 


1 6 .  Mechanic  assignment  0 


17.  Logon  graphics 
great! 


18.  Process  was 
excellent 


6  5 
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Acoendix  E:  Idea  Voting  Results  TContinued 


1 .  T.O.  graphics  insufficient 


2.  T.O.  presentation 


3.  Review  previous  write-up 


4.  Aircraft  operation  status 


5.  Aircraft  identification 


Mean  SIB  Total  n 


8.9  1.2  62 


8.9  1.4  62 


8.4  1.3  59 


8.4  I  1.5  I  59 


8.4  1.7  59 


6.  Bar  graph  colors  need  to  be  explained.  8.3  1.9  58 

ex.  red-  behind  schedule,  (legend) 


7.  Get  ready  to  do  maintenance  7.9  0.4  55 


7.9  1  1.1  55 


8.  Planning  interface  requirement 


9.  Should  be  able  to  return  to  T.O..  task  7.9  1.1  55 

when  documenting  maint. 


10.  Group  writeups  with  work  areas  7.6  1  53 


11.  173  cards  need  to  be  shown  on  screen 
as  a  number 


12.  Support  equipment  7.3  1.3  51 


13.  Perform  maint 


14.  Age  equipment  backorder  unclear 


15.  Closing  out  not  clear  from  task  to 
task 


16.  Mechanic  assignment 


17.  Logon  graphics  great! 


18.  Process  was  excellent 
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Appendix  F;  Categorized  Rapid  Prototype  Evaluation 


Appendix  F  is  the  product  of  the  original  categorization  process.  It  contains  the  ideas  and 
comments  generated  by  the  users  placed  into  predetermined  categories.  The  users,  as  a  group, 
took  the  generated  ideas  and  placed  them  into  the  categories.  Note  that  the  categories  are 
eliminated  in  table  1  because  they  did  not  provide  valuable  information  for  the  human  factors 
engineers  developing  ITI-ALC. 

USER  EXPECTATIONS 

1.  AIRCRAFT  IDENTIFICATION 

•  are  these  aircraft  tail  numbers?  which  is  assigned  to  me? 
highlighted  aircraft  signify  managers  aircraft. 

•  need  bldg  location  i.e.  bldg  number  and  bay 

•  Selection  of  aircraft  should  be  consistent  with  other  selections.  Single  mouse  click 
without  watch  symbol  confusing.  Double-clicking  caused  work  form  to  update. 

•  identify  which  aircraft  belongs  to  which  aircraft  manager,  this  info  would  be  useful  in 
can  items,  (read  only) 

•  provide  history  of  aircraft  configuration 

-  which  tctos  have  been  complied  with? 

-  what  special  work  has  been  performed? 

2.  TO  Presentation 

•  need  capability  to  mark  last  completed  step  (e.g.,  bookmark) 

•  warnings  before  starting  tasks  are  very  helpful. 

•  to  ensure  proper  recognition  of  icons  computer  classes  for  all  mechanics  a  must. 

3.  AIRCRAFT  OPERATION  STATUS 

•  status  bar  is  difficult  to  read,  would  rather  see  a  time  line  of  operation  showing 
completion 
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4.  BAR  GRAPH  COLORS  NEED  TO  BE  EXPLAINED.  EX.  RED-  BEHIND  SCHEDULE, 
(Legend) 

5.  SUPPORT  EQUIPMENT 

•  need  to  show  part  number 

•  fed.  stock  number,  and  nomenclature. 

6.  TO  graphics  insufficient 

•  graphic  donest  show  gap  tol. 

•  picture  of  rack  not  sufficient 

•  graphics  should  be  displayed  full  view  on  entry  and  be  able  to  zoom  in  on  specific 
areas 

•  need  more  than  one  view  on  locator  area  around  work  needs  to  be  more  detailed 

•  add  menu  capability  for  exploded  views. 

SCREEN 

1.  CLOSING  OUT  NOT  CLEAR  FROM  TASK  TO  TASK 

•  wording  unclear  could  not  find  sign  off 

•  have  an  exit  button  to  log  out 

2.  LOGON  GRAPHICS  GREAT! 

•  Aircraft  fly-by 

3.  PERFORM  MAINT 

•  like  acft  locator 

•  have  capability  to  reinitiate  alert  from  the  menu  or  icon  (step  7). 
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NAVTGATTON 


1.  MECHANIC  ASSIGNMENT 

•  did  not  know  clicking  on  mechanic  name  was  way  of  selecting  name 

2.  PLANNING  INTERFACE  REQUIREMENT 

•  need  interface  with  planning  organization  as  with  problems  in  process  such  as  tcto 
compliance,  e.g.,  cams  indicates  a  tcto  being  c/w  when  actually  it's  not 

•  need  interface  with  engineering  for  202  requests  and  also  afto  22  requests 

3.  SHOULD  BE  ABLE  TO  RETURN  TO  T.O.  TASK  WHEN  DOCUMENTING  MAINT. 
TERMINOLOGY 

1.  REVIEW  PREVIOUS  WRITE-UP 

•  needs  to  be  reworded  to  reflect  "previous"  in  lieu  of  o  level 

•  how  do  you  get  to  previous  write-ups 

•  bad  menu  choice  o-level  history 

•  have  write  ups  come  up  before  job  is  started 

•  spell  out  "o  level"  to  organization 

•  don’t  abb.  menu  items 

Errors 

1. 173  CARDS  NEED  TO  BE  SHOWN  ON  SCREEN  AS  A  NUMBER 

•  inspection  code  should  be  predetermined,  not  an  option.  Critical  requires  2, 
noncritical  requires  1 

•  Need  to  allow  one  signature  box  for  each  person  signing  the  "card." 

T.EARNTNG 
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SYSTEM  CAPABTLTTY 


1.  GET  READY  TO  DO  MAINTENANCE 

•  need  to  reflect  between  age  and  test  equipment 

•  need  further  instruction  on  how  to  order  equipment 

•  need  to  show  ordering  status 

•  can't  perform  task  unless  equipment  is  available 

•  on  parts  source  what  is  none 

2.  AGE  EQUIPMENT  BACKORDER  UNCLEAR 

•  has  equipment  on  backorder  already  been  ordered  or  does  mechanic  need  to  order? 

3.  GROUP  WRITE  UPS  WITH  WORK  AREAS 

•  Group  tasks  by  physical  location. 

MISCELLANEOUS 

1.  PROCESS  WAS  EXCELLENT 

•  scenario  was  excellent 
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Appendix  G:  Histograms  for  the  Usability  Questionnaire  Items 


USER  EXPECTATIONS 

Question  1:  Overall,  use  of  the  system  was  <Frustrating  (1)  to  Satisfying  10)> 


Question  2:  Overall,  use  of  the  system  was  <Difficult  (1)  to  Easy  (10)> 


Question  3:  The  needs  of  both  experienced  and  inexperienced  users  was  considered 


<Never  (1)  to  Always  (10)> 


Question  3 


Never  Rating  Scale  Always 
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SCREEN 


Question  4:  The  characters  on  the  computer  screen  were  <Hard  to  read  (1)  to  Easy 
to  read  (10) 


99 


Question  5:  The  sequence  of  information  on  the  screen  was  <Disorganized  (1)  to 
Organized  (10)> 


1  23456789  10 


Disorganized  Rating  scale  Organized 


NAVIGATION 


Question  6:  Logging  into  the  system  and  starting  the  task  was<Difficult  (1)  to  Easy 

(10)> 


Question  6 


Question  7:  Navigation  among  items  (e.g.  fill-ins)  on  the  screen  was  <Illogical  (1)  to 


Logical  (10)> 


123456789  10 


lltagical  Logical 
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0 


Question  10:  Logging  off  the  system  was  <Difficult  (1)  to  Easy  (10)> 
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TFRMTNOT.OGY 


Question  11:  The  terminology  used  throughout  the  system  was  <Hard  to 
understand  (1)  to  Easy  to  understand  (10)> 


#  Responses 


Question  12:  In  relation  to  the  work  you  do,  the  use  of  terminology  was 


Inconsistent  (1)  to  Consistent  (10)> 


Question  12 


123456789  10 

Inconsistent  Rating  Scale  Consistent 


107 


ERRORS 


Question  14:  Correcting  mistakes  was  <Difficult  (1)  to  Easy  (10)> 
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Question  15:  The  number  of  ITI-ALC  system  failures  encountered  was  <#  of 
failures> 


Question  15 


Question  16:  Number  of  error  messages  encountered  was  <#  of  errors> 


Question  16 


Number  of  Errors 

Rating  Scale 
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Question  17:  Recovering  from  errors  was  <Easy  (1)  to  Difficult  (10)> 


2 


LEARNING 


Question  18:  Learning  how  to  use  the  system  was  <Easy  (1)  to  Difficult  (10)> 


1  23456789  10 


ESSy  Rating  Scale  Difficult 
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Question  19:  Remembering  the  names  and  use  of  commands  was  <Difficult  (1)  to 


Easy  (10)> 


Question  19 


”■”•.'■■'-1 - 1 - 1 - 1 - 1 - 1 - 1 - : - : - 

1  23456789  10 


Difficult  Rating  Scale  Easy 
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Question  20:  Exploration  of  system  functions  was  <Rigid  (1)  to  Flexible  (10)> 


Question  20 


^‘9’^  Rating  Scale  Flexible 
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Question  21:  The  task  flow  between  the  real  world  and  the  system  seemed  to  be  the 


<Different  (1)  to  Same  (10)> 


SYSTEM  CAPABILITIES 


Question  22:  Response  time  from  the  system  was  <Slow  (1)  to  Fast  (10)> 
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Question  23:  Overall,  the  system  was  <UnreIiable  (1)  to  Reliable(10)> 


ITT-ALC 


Question  24:  What  did  you  particularly  like  about  ITI-ALC? 

Total  Nvimber  of  Respondents  (N):  7 

Number  of  Responses  to  This  Question  (n):  7 


1 .  my  grand  kids  will  love  it 

2.  WORKING  PROCESS 

3.  It  gives  basically  all  information  which  is  needed  to  accomplish  a  work  task. 

4.  something.to.help.mech 

5.  Overall  sequence  was  great.  Flowing  from  one  step  to  another,  fairly  easy  with  minor 
problems. 

Research  group  personnel  very  helpful  and  friendly. 

Fovmd  several  areas  where  modifications  needed  to  be,  and  changes  seemed  very  easy. 

6.  Flow  of  work  process.  Easier  to  track  work  and  parts 

7.  STEP  BY  PROCEDURE 
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Question  25:  What  in  particular  did  you  NOT  like  about  ITI-ALC? 


Total  Number  of  Respondents  (N) :  7 

Number  of  Responses  to  This  Question  (n):  6 

1. 1  will  be  dead  before  I  get  to  use  it 

2.  No  instructions  up  front  to  navigate 

3.  nothing 

4.  Some  commands  were  a  little  misleading.  Some  steps  hard  to  follow.  Not  enough 
information  on  views  of  work  areas 

5.  overall  system  needs  to  be  more  user  friendly.  More  T.O.  information  needs  to  be  installed 

6.  NOTHING 
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Question  26:  What  recommendations  would  you  make  for  improving  ITI-ALC? 


tt 


T otal  Number  of  Respondents  (N) :  7 

Number  of  Responses  to  This  Question  (n):  7 


1.  try  to  make  it  more  user  friendly 

2.  NEED  MORE  SCHOOLING  ON  COMPUTER  SYSTEMS 

3.  Make  it  more  user  friendly. 

4.  HAVE  A  FEW  MORE  STUDIES 

5.  Provide  user  friendly  computer  classes  to  ensure  continuity  between  operators. 

6.  More  input  from  mechanics. 

7.  HAVE  A  PRESENTATION  ON  BASIC  COMPUTER 
SKILLS 


121 


GSS 


Question  27:  I  would  use  a  Group  Support  System  to  evaluate  another  prototype 


Question  27 


Strongly  Agree  Rating  scale  Strongly  Disagree 
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#  Responses 


Question  28:  The  GSS  helped  me  to  evaluate  the  rapid  prototype  faster  than  if  I 


didn't  use  it. 


Question  29:  The  GSS  was  effective  in  helping  me  evaluate  the  rapid  prototype. 


Question  29 


strongly  Agree  Strongly  Disagree 
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Appendix  H: 


Scenario  Screen  Sequence  and  Observation  Notes 


Appendix  H  presents  the  screens  seen  by  the  users  during  the  evaluation  of  the  rapid 
prototype.  The  screen  is  at  the  top  of  each  page,  with  related  comments  by  the  observers  beneath 
it.  The  listing  is  an  aggregation  of  all  observer  notes  for  that  screen. 


•Didn’t  pick  a  name 

•No  selection  of  role  before  ok 


125 


•Question  about  whether  to  input  name.  Provided  menu  not  clear 
Wasted  step—go  ahead  list  people  out 
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Cfidk  on  a  i«yhigitled  Master  SehcMiiibJ 


•Double  click  hides  schedule— A  general  problem  with  minimize  and 
maximize  without  executing  commands  (S/W  problem) 

•Individual  was  a  novice  user,  unfamiliar  with  general  computer  procedures 


•Was  able  to  learn  some  of  the  interface  thru  trial  and  error  (e.g.  logout) 


•User  selected  HUD 


•User  canceled  window 


•Would  be  nice  to  have  major  jobs  on  one  graph  along  with  pertinent  information 
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•Tim  T.  Toolman  ended  up  blank 


•Searching—cXioks  on  numbers  17,  then  18,  then  clicked  on  Dan  Carlson,  then 
Tim  T.  Toolman 


•User  tried  editing  schedule  without  closing  dialog  box 
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•Difficulty  with  mechanic  list  box 

•Hidden  functions  under  name  or  task  isn’t  clear— ended  up  closing  application 
trying  to  find  menu  item  in  upper  menu  box 
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m-AllC^lMasterSiahSauW^ 


'^1  Eilc  Work  Oper.  Prepare  for  Maint  iechjnfo  issk  Graphic  M^sc,  Help 


DeceH^f ' 

12/10, 

^ai?f 

ii 

ii 

WBH 

iM 


J^$  for  A73^C}J^F 


~  As%ied 

"^Tov ;: 


Ibvembt 


m 

M 

m 

go 

m 

m 


<K> 


Ifeffe^placeHUty 


iSQy^^FI^ 


liy;^ijOvSfK^-Rlgl| 


IfiO^hoiSLifi 


jQpe^aSohalCNgq 


MulthSkilled  Mechanics 


This  Assignments  screen  demonstrates  the  ITI-ALC 
concept  of  having  multhskilled  mechanics.  Having 
mechanics  who  are  certified  across  differing  functions 
will  enable  broader  user  of  resources  and  reduced  flow 
days 


tr 


Recommended 


PACs: 


Mechanic(s):  Dan  Carlson 


Avionics,  Removing  Flight 
Controls,  Electrician 


Grade: 

Grade  10 

""V 
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ITI-ALC  “  [Master  Schedule] 


a 

Elffi  WorkOper.  Prepare  for  Maint  Tech  info  Task  Graphic  MIsc.  Help 

pi 
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nS 

•User  kept  looking  for  the  words  “Sign  off’ 


•Looked  at  Logout  about  3  times  and  finally  selected  it 


•User  was  uneasy  about  using  menu  selection 
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•Asked  what  status  meant 

-thought  but  not  sure  it  was  completion 


•Review  write  ups 

-began  task  before  receiving  tech  data 
-began  task  before  reviewing  Support,  equipment 
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OTrAlX  A  [WORK  QPERAJIONS  LIST] 


H  file  Work  Oper,  Prepare  for  Main!  Tech  info  Task  Graphic  Misc.  Help 


Aircraft  Tail  Number:  A7356(C) 


U4 

Hbm 


$L 

erk 


Replace  FCR  Equipment  Rack  Left  Support  Assy 
Remove  FCrI 


Install  FCR  i 
Overhaul  Door 


100 


Sched-StartCSaft^ 

D^e  r  Daie  7 


11/28/95 


Process  and  Terminology  Coordination 


111/28/95 

bs/ss 


The  Work  Operations  List  presents  to  the  maintainer  a 
prioritized  list  of  Work  Operations  and  Tasks  to  be 
completed.  In  the  ITI-ALC  system^  this  list  is  organized 
by  an  expert  system  to  display  Work  Operations  in 
order  based  on  parts  availability^  time  to  complete,  and 
the  logic  of  doing  certain  jobs  before  others.  Further, 
this  Work  Operations  List  will  make  use  of  the  ITFALC 
recommended  common  terminology.  For  example,  what 
is  known  in  the  AS-IS  world  as  a  1 73  card,  will  be  known 
in  this  TO-BE  environment  as  a  Work  Operation. 


P/95 


mm^ 


5UZ 
100^ 
OX 

OX 
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•Wasn’t  sure  what  “O”  meant  in  0-level  on  the  menu 
-suggested  to  not  abbreviate 

•0-level  History  not  obvious  to  review  previous  write-ups 
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ITI-ALC  -  [ORGANIZATIONAL  LEVEL  HISTORY] 


»?|  File  Work  Oper.  Prepare  for  Maint  Techinfo  jask  Graphic  Misc.  Help 


AIRCRAFT  TAIL  NO:  A7356(C) 


S  YSTEM : 


TORPECTiyE'-^IONi^lf 

10/24/95 

MFL  004  r 

Replace  Antenna 

Yes 

MFL  084 

Replace  DMT 

No 

6/24/95 

MFL  004 

Replace  Antenna 

No 

4/6/95 

MFL  004 

Replace  Antenna 

|No 

1/11/95 

MFL  004 

Wire  Repair 

No 

I 


•User  wants  previous  write  ups  to  come  up  automatically  when  receiving  the  job 
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'>  m-ALC-fORGAMIZ^TIONAI-lJEVELHlSTdFr«l»^^^  ^ 

ami 

IQ 

File  Work  0 per.  Prepare  for  Maint  Tedi|nfo  lask  Graphic  Mlsc,  Help 

AIRCRAFT  TAIL  NO:  A7356(C)  SYSTEM: 

II 

ppc^PAN^.;]:^ 

CORRECTIVE  ACTION  ; 

MreAT/RECURr-'i., 

10/24/95 

MFL  004 

Replace  Antenna 

Yes 

8/8/95 

MFL  084 

Replace  DMT 

No 

6/24/95 

MFL  004  I 

Replace  Antenna 

No 

4/6/95 

MFL  004 

No 

1/11/95 

Q  Q  Data  Sharing  Among  All  Levels  of  Maintenance 

-■  1 - ' 

The  0-LeveI  History  screen  gives  the  maintainer 
visibility  into  previous  actions  performed  on  a  certain 
aircraft  at  the  flightline.  This  gives  him/her  an 
advantage  in  knowing  Information  such  as  repeating 
and  recurring  faults. 
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ITI-ALC  -  [WORK  OPERATIONS  LIST| 


k  File  WorkOper.  Erepare  for  Maint  Tech  Info  Task  Graphic  Misc.  Help 


Aircraft  Tail  Numbs 


Receive  Tech  Info 
gupp  Equip  Status 
Kits  Status 


1;  -  » 

;.Std.r 

'Hoiwt 

ii-Parts 

-i^Avai::,; 

Si^edSl^. 

ij3 .  y  Name 

.i;'  , 

»  Replace  FCR  Equipment  Rack  Left  Support  Assy 

4 

100 

11/28/95 

11/28/95 

50^ 

Remove  FCR  Equipment  Rack  Left  Support  Assy 

2 

100 

11/28/95 

11/28/95 

loo:^ 

Install  FCR  Eauiomenl  Rack  Left  Subobrl  Assy^ ; . .  „ 

,  2 

100 

11/28/95 

11/28/95 

mttz : , 

^  Overhaul  Door  1202 

4 

80 

11/30/95 

11/30/95 

0% 

}Ries«.!le#itTadK" piocedlurB,S;i^-  : 
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m 


m-jax:g  IWORK  OreRATIONS 


■mi  File  Work  Oper.  Prepare  for  Maint  Tech  Info  lask  Graphic  Misc,  Help 


RECEIVE. 


Receiving  All  Technical  Order 
Information  for  FOR  Equipment  Rack, 
Left  Support  Assembly,  Installation 
(94-61-06) 


0% 


TjTiyynng^ 


I 


100% 


Prett  Ta^ 


Aya3;;' 

S|<pNGd'S^'rt; 

m 

11/28/95 

11/28/95 

50^ 

11/28/95 

11/28/95 

100% 

11/28/95 

11/28/95 

0% 

1 

11/30/95 

11/30/95 

Automated  and  integrated  Technical  and  Diagnostics  Information 


The  maintainer  will  have  the  capability  to  download  all  the 
technical  information  necessary  to  complete  his/her  work.  This 
will  allow  mechanics  to  get  the  directions  and  information  they 
need,  at  whatever  level  of  detail  is  required,  instantaneously. 
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ITI-ALC  -  [SUPPORT  EQUIPMENT  STATUS] 


‘  File  Work  Oper.  Prepare  for  Maint  Tech  Info  I^ssk  Gmphlc  Help 


ISSUED  TO:  On  Hand 


'-i  I 

imii 

A 

Each 

^920-01 -202-6S95DQ 

A/E  24T-169 

AVAILABLE 

A 

Each 

Each 

5220-00-192-1488 

AVAILABLE 

'■  .  i 

;4920-01-034-8670DP 

y  i, ■  ty?’-  ^.5  Jf  Sivill 

AN/ASM-497 

BACK-ORD. 

JpIPST 


•How  to  order  was  still  not  readily  apparent 
•Still  looking  for  words 

•Had  cursor  on  Equipment  several  time  but  wasn’t  sure  it  was  right  and  was 
•uneasy  to  try 

•Separate  between  AGE  and  Test  equipment.  Otherwise,  don’t  know  where  to  find 
AGE 

•Should  go  ahead  and  say  whether  or  not  a  job  is  supportable-give  reasons, 
give  delay  times 
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•Source  of  “none”  confusing 

•Still  need  status  of  ordered  parts/equipment  or  else  task  can’t  be  done 
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imLC- [INVENTORY  PARTC  &^l^  . 


File  WorkOper.  Prepare  for  Maint  Techiofo  Issk  Graphic  Mlsc,  Help 


JOB  NAME:  Replace  FCR  Left  Equipment  Rack  Support  Assembly 


"omM: 

(FpQ’K 

:QT^?Sk 

AVAIL. 

1  S  Acquire  Parts  &  Visibility  Into  Parts  Availability  I 

T 

The  Inventory  Parts  &  Kits  Status  screen  gives  the 

A 

4 

maintainer  visibility  into  what  parts  are  required  for  a 
Work  Operation,  and  the  current  availability  of  these 
parts  in  the  Kit.  The  maintainer  can  also  order 
necessary  parts  right  from  this  screen. 

In  Kit 

4 

A 

liWi^l 

In  Kit 

1 

Each  :BRM3  FCR  EQUIPMENT  iNone 

In  Stock 

RACKLEFT  i 

1 

SUPPORT /«SY 
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ITI-ALC  -  [WORK  OPERATIONS  LIST] 


File  Work  Oper.  Prepare  for  Maint  Tech  info  Task  Graphic  Misc.  Help 


Aircraft  Tail  Number:  |A7356(C)  |H 


:\'SW  X  Sai8d;St^J:g|i^' 

;.Hdu9r&  ;''"Aya8"^  .t;- 


: Overhaul  Door  1202 


4 

100 

11/28/95 

11/28/95 

2 

100 

11/28/95 

11/28/95 

too 

11/28/95 

11/28/95 

4 

80 

11/30/95 

11/30/95 

•Highlighted  line  below  Install  FCR...then  tried  to  begin  task 
•Highlighted  several  other  tasks,  user  was  not  sure  what  to  do 
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CR  Equipment  RacK,'  Left  Support  Assemblyi-lHstaliaflon  (94-61’ 


<#.|  File  Work  Oper.  Prepare  for  Maint  Tech  (nfo  Task  Graphic  Misc.  Help 


INPUT  CONDITIONS 


PERSONNEL  RECOMMENDEI 
1  Technician 

SUPPORT  EQUIPMENT: 
Torque  Wrench  (torque  ranc 

SUPPLIES  (CONSUMABLES): 


NAME'X;  • : , 'f ' '  /  'a  "  '  ■ 


Shim 


Shim 


User  Technical  Information  Presentation  System 


The  ITPALC  System  has  an  integrated^  simple-to-use 
presentation  system  for  all  the  Technical  Information 
the  maintainer  requires.  This  presentation  system  has 
the  flexibility  to  provide  the  user  with  extra  information 
such  as  Theory  of  Operations,  Illustrated  Parts 
Breakdown,  and  Locator  Graphics.  Further,  it  enables 
the  maintainer  to  request  additional  information 
(Engineering  Assistance  Request],  make  suggestions 
as  to  the  improvement  of  the  Technical  Information,  etc. 


Shim 

16E31 04-25 

Eac 

■ 

Shim 

16E31 04-27 

1 _ d 

U 

Eac 

■ 

REFERENCE  MATERIALS: 

Feeler  Gage  -  Measure  gaps  between  aircraft  structure  and  rack 


la  tee  the^Hequmd  C^dtoqri»> *-->>5-'  r-f  ■- 


•Double  clicked  to  see  kit  info 


•Knowing  exactly  when  you  are  ready  to  start  the  task  is  unclear 

•Screens  with  no  action  — no  one  knows  where  to  proceed  unless  directed  to 
on  screen 
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•Double  clicked  several  times 


•Tough  to  determine  navigation  sequence 
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•Asked,  “is  this  the  size  of  the  picture?” 


Could  not  find  how  to  indicate  elect,  bonding  complete 


ITI-ALC  “  [FCR  Equipment  Rack,  Left  Support  Assembly,  installation  {94-61-06]. 


H  File  Work  Open  Prepare  for  Maint  Techjnfo  lask  Graphic  Misc.  Help 


IebhI 

HBSO 

^1  - 


3-  Position  rack  into  aircraft. 


A.  Install  four  washers  and  four  screws. 

5.  Install  four  mount  screws. 

6.  Using  feeler  gage,  measure  gap  between  aircraft  structure  and  rack. 


•T.O.  error— graphic  doesn’t  show  the  tolerances 
•User  unfamiliar  with  use  of  scroll  bars 
•Skipped  hot  spot,  didn’t  look  at  the  access  door 


4 


152 


153 


•Clicked  on  step  8  to  change  highlighting 

•“Back”  label  on  locator  and  Full  Screen  may  be  confusing 

•Looking  under  “Graphic”  menu  bar  to  get  back. 

•couldn’t  get  notes/cautions  from  icons  or  menus 
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ITI-ALC  -  [WORK  OPERATION  FORM] 


File  Work  Qper,  Prepare  for  Main!  Tech  jnfe  j^sk  Graphic  Help 


Aircraft  Tail  Number:  A7356(C) 


System  : 


Fire  Control  Radar 


Description:  Replace  the  Left  Support  Assembly  for  the  FOR 

Equipment  Rack  (94-61-06). 


Problems  Encountered 
During  Work  Operation: 


Inspection  Code: 


Signature: 


MiMggj 


•Unfamiliar  with  menu,  didn’t  go  to  file  to  find  logout 

•Went  back  and  forth  from  “Work  oper.  to  “Help”  on  menu  3  times 

•B  card  should  be  displayed  without  ability  to  change  to  A  card  (1  inspector) 

•No  note  that  the  last  formJs  the  173.  Looked  for  “173  card”  under  above 
menu  items 

•Not  worked  as  a  “173  card”— confused  with  the  first  view  of  the  form,  from 
the  menu  which  had  no  place  for  typed  inputs 
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s 

SBp 

B 

File  WorkOper.  Prepare  for  Maint  Tech  info  Task  Graphic  M*sc.  Help  W4 

Aircraft  Tail  Number: 

Bi  Electronic  Signatures  ~  1 1 

System  : 

The  Work  Operation  Form  presents  to  the  maintainer  the 

Description: 

information  he/she  needs  regarding  a  Work  Operation. 

When  this  Work  Operation  is  completed,  the  maintainer 
then  signs  off  that  the  job  is  done  via  an  electronic 
signature. 

Problems  Encountered 
During  Work  Operation: 

li^ 

Inspection  Code: 
Signature: 


A:  2  People 


i 


jS.eiid  I 


•People  not  used  to  clicking  on  an  empty  box.  Short  time  to  realize. 

•Comment  block  is  good  but  may  be  checked  elsewhere,  (e.g.  why  wasn’t  task 
equipment  available) 


4 
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Aircraft  Tail  Number.  A7356(C) 


# 


« 
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•Accidentally  logged  in  as  mechanic 
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ITI-ALC  -  [Depot  Floor] 


|ia  File  Work  Qper.  Prepare  for  Maint  Tech  Info  lask  Graphic  Mlsc.  Help 


•Double  clicked  which  brings  up  Task  edit 

•Double  click,  without  cursor  change,  caused  form  update  automatically 
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ITI-ALC  ”  (Master  Schedule] 


^  File  WorkOper.  Prepare  for  Maint  Tech  Info  lask  Graphic  M>sc,  Help 
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<^  _ 


IBpkaHSSffKSckoul 


WailiKtoh  A 


PreFiirf^Servicfe 


DEyVERV!! 


■'.:»  sVrrC:;  ? 


Work  Operation  Completion  Notice 


Work  Operation: 

Replace  FCR  Left  Equipment  Rack 
Support  Assembly 

Completed  on  A/C: 

A7356rCl 


P 

B 

I 

p; 

•This  form  should  be  model 


•Skipped  receiving 
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ITI-ALC  -  IWORK  OPERATION  FORM  -  COMPLETED] 


WorkQper.  Prepare  for  Maint  Tech  jnfo  lask  Graphic  Misc.  Help 


Aircraft  Tail  Number: 

System  : 
Description: 


A7356(C) 

Fire  Control  Radar 

Replace  the  Left  Support  Assembly  for  the  FOR 
Equipment  Rack  (94-61-06). 


Problems  Encountered 
During  Work  Operation: 

Inspection  Code: 
Signature: 


Start  Time: 
Stop  Time: 


In  one 


|A:  2  People 


111/28/35  8:07:00  AM 


111/28/95  11:40:00  AM 


5 

ESI 

^  ITItALC - fWORK OI^RATION  FORM 

HU 

B 

File  Work  Oper.  Prepare  for  Malnt  Tech  info  Task  Graphic  Miisc.  Help 

Aircraft  Tail  Number:  I 

Performance  Metrics  Based  on  Actual  Data 

■ 

System  : 
Description: 

Problems  Encountered 
During  Work  Operation: 

Inspection  Code: 
Signature: 


o 


The  completed  Work  Operation  Form  that  the 
Aircraft  Manager  reviews  has  the  actual 
performance  time  for  that  Work  Operation. 
Traditionally,  the  cost  of  maintenance  work  has 
been  based  on  standard  hours  assigned  to 
each  Work  Operation,  but  these  standard  hours 
are  rareiy  adjusted  for  specific  stuations.  The 
ITI-ALC  system  allows  for  adjustment  of  hours 
and  cost  of  maintenance  work  based  on  real 
data. 


liOKjtl 


start  Time: 
Stop  Time: 


11/28/95  8:07:00  AM 


11/28/9511:40:00  AM 


»ii 
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IThALC  “  [Master  Schedule] 


B 

file  Work  Oper.  Prepare  for  Maint  Tech  info  Task  Graphic  M»sc.  Help 
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•Functions  under  tasks,  names,  calendar  aren’t  clear.  Wanted  to  see  completion 
times  on  the  graph  but  couldn’t  see  them 
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Appendix  I;  Human  Use  Release  Form 


Informed  Consent  Document 
for 

New  Technologies  for  Maintenance 
and  Logistics  Information  System  Studies 


1.  Nature  and  Purpose:  I  have  been  asked  to  vohmteer  as  a  subject  in  the  project  named  above.  The 
purpose  of  these  studies  is  to  test  and  evaluate  candidate  technologies  for  maintenance  and  logistic 
information  systems.  The  studies  will  help  to  determine  the  best  way  to  design  various  features  of 
the  systems.  The  present  study  is  designed  to  perform  an  evaluation  of  a  rapid  prototype  developed 
for  the  Integrated  Technical  Information  for  the  Air  Logistics  Center  (ITI-ALC)  program.  For  this 
study,  one  session  with  a  duration  of  one  day  is  required. 

2.  Experimental  procedures:  Each  user  will  be  required  to  evaluate  the  ITI-ALC  rapid  prototype  and 
document  the  comments  and  findings  of  their  evaluation  using  a  group  support  system. 

3.  Discomfort  and  risk:  I  understand  that  participation  in  the  study  will  not  involve  risks  greater 
than  those  encoimtered  in  performing  my  normal  duties.  Working  conditions  will  be  similar  to  those 
encountered  in  my  normal  duties. 

4.  Precautions  for  female  subjects:  The  study  does  not  involve  risks  specific  to  females.  Therefore, 
no  special  precautions  are  required. 

5.  Benefits:  There  is  no  direct  financial  or  medical  benefits  to  the  subject.  However,  participation 
provides  the  opportunity  for  the  subject  to  contribute  to  the  design  and  development  of  future  Air 
Force  information  systems. 

6.  Entitlements  and  confidentiality: 

a.  Records  of  participation  in  this  study  may  only  be  disclosed  according  to  federal  law, 
including  the  Federal  Privacy  Act,  5  U.S.C.  552a,  and  its  implementing  regulations. 

b.  I  understand  that  my  entitlements  to  medical  and  dental  care  and/or  compensation  in  the 
event  of  injury  are  governed  by  federal  laws  and  regulations,  and  that  if  I  desire  further  information  I 
may  contact  the  base  legal  office. 

c.  If  an  unanticipated  event  (medical  misadventure)  occurs  during  my  participation  in  this 
study,  I  will  be  informed.  If  I  am  not  competent  at  the  time  to  understand  the  nature  of  the  event, 
such  information  will  be  brought  to  the  attention  of  my  next  of  kin. 
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d.  The  decision  to  participate  in  this  research  is  completely  voluntary  on  my  part.  No  one  has 
coerced  or  intimidated  me  into  participating  in  this  program.  I  am  participating  because  I  want  to. 
Capt  Allen  Gwartney  AFIT/LAA,  57777  x  2244  has  adequately  answered  any  and  all  questions  I 
have  about  this  study,  my  participation,  and  the  procedures  involved.  I  understand  that  Capt  Allen 
Gwartney  will  be  available  to  answer  any  questions  concerning  procedures  throughout  this  study.  I 
understand  that  if  significant  new  findings  develop  during  the  course  of  this  research  which  may 
relate  to  my  decision  to  continue  participation,  I  will  be  informed.  I  further  imderstand  that  I  may 
withdraw  this  consent  form  at  any  time  and  discontinue  further  participation  in  this  study  without 
prejudice  to  my  entitlements.  I  also  understand  that  the  medical  monitor  of  this  study  may  terminate 
my  participation  in  this  study  if  he  or  she  feels  this  to  be  in  my  best  interest. 


Volunteer  Signature  and  SSN 


Date 


Investigator  Signature  and  SSN  Date 


Witness  Signature  and  SSN 


Date 


A 
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Appendix  J:  List  of  Acronyms 


AFIT 

Air  Force  Institute  of  Technology 

AFTO 

Air  Force  Technical  Order 

AGE 

Aerospace  Ground  Equipment 

AL 

Armstrong  Laboratory 

ALC 

Air  Logistics  Center 

BPI 

Business  Process  Improvement 

CALS 

Continuous  Acquisition  Life-Cycle  Support 

DoD 

Department  of  Defense 

FCR 

Fire  Control  Radar 

GSS 

Group  Support  System 

GUI 

Graphical  User  Interface 

HCI 

Human  Computer  Interface 

IDEF 

Integrated  Definition  Methodology 

ITI-ALC 

Integrated  Technical  Information  for  the  Air  Logistic  Centers 

Mhz 

Mega-Hertz 

PDM 

Programmed  Depot  Maintenance 

RF 

Radio  Frequency 

SRA 

System  Research  and  Application  Corporation 

T.O. 

Technical  Order 

TCTO 

Time  Compliance  Technical  Order 
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